2024-04-02T04:26:52.474 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-04-02T04:26:52.475 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_teuthology@teuthology --name teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi --description orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} --archive /home/teuthworker/archive/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634849 -- /home/teuthworker/archive/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634849/orig.config.yaml 2024-04-02T04:26:52.517 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-02T04:26:52.605 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634849 branch: squid description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} email: ceph-qa@ceph.com first_in_suite: false job_id: '7634849' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 6cec793d9073f0594e2364b9b687ecaf22424cca ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 6cec793d9073f0594e2364b9b687ecaf22424cca selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 workunit: branch: squid sha1: 6cec793d9073f0594e2364b9b687ecaf22424cca owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - host.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - host.b - osd.4 - osd.5 - osd.6 - osd.7 seed: 6530 sha1: 6cec793d9073f0594e2364b9b687ecaf22424cca sleep_before_teardown: 0 subset: 52/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_6cec793d9073f0594e2364b9b687ecaf22424cca/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 6cec793d9073f0594e2364b9b687ecaf22424cca targets: smithi053.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB6lGwmTurR8jDyMJ7pg92j/x13b4nh/4eyDxRLVKZZPOw1+Or1d1EcRGjSwjo1rtDinHJ2GPyuDYw7MuajOFjY= smithi122.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPH70r4gDZkVgtnf+XQEzpooFGq8G2FD8nASXwlDtS7Z7n55T3x+t3yEDXeqiB1V862PNVOszCjW6l+gUgJiLiE= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 roleless: true - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - vip.exec: all-hosts: - systemctl stop nfs-server - cephadm.shell: host.a: - ceph fs volume create foofs - cephadm.wait_for_service: service: mds.foofs - cephadm.shell: host.a: - ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2 - ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake - while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done - vip.exec: host.a: - mkdir /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done - echo test > /mnt/foo/testfile - sync - parallel: - upgrade-tasks - workload-tasks - vip.exec: host.a: - umount /mnt/foo - cephadm.shell: host.a: - ceph nfs cluster ls | grep foo - ceph nfs export ls foo --detailed - rados -p .nfs --all ls - - cephadm.shell: host.a: - 'set -ex [[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]] ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-04-01_21:08:08 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph mgr module enable nfs --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - cephadm.wait_for_service: service: nfs.foo user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - exec: host.a: - cd /mnt/foo && dbench 5 -t 600 || true - umount /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done - cd /mnt/foo && dbench 5 -t 5 2024-04-02T04:26:52.605 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_6cec793d9073f0594e2364b9b687ecaf22424cca/qa; will attempt to use it 2024-04-02T04:26:52.606 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_6cec793d9073f0594e2364b9b687ecaf22424cca/qa/tasks 2024-04-02T04:26:52.606 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-02T04:26:52.608 INFO:teuthology.task.internal:Checking packages... 2024-04-02T04:26:52.633 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '6cec793d9073f0594e2364b9b687ecaf22424cca' 2024-04-02T04:26:52.633 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-02T04:26:52.633 INFO:teuthology.packaging:ref: None 2024-04-02T04:26:52.633 INFO:teuthology.packaging:tag: None 2024-04-02T04:26:52.633 INFO:teuthology.packaging:branch: squid 2024-04-02T04:26:52.634 INFO:teuthology.packaging:sha1: 6cec793d9073f0594e2364b9b687ecaf22424cca 2024-04-02T04:26:52.634 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-02T04:26:52.884 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1979.g6cec793d 2024-04-02T04:26:52.885 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-02T04:26:52.891 INFO:teuthology.task.internal:no buildpackages task found 2024-04-02T04:26:52.891 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-02T04:26:52.911 INFO:teuthology.task.internal:Saving configuration 2024-04-02T04:26:52.925 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-02T04:26:52.932 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-02T04:26:52.956 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634849', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-02 04:21:54.924375', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB6lGwmTurR8jDyMJ7pg92j/x13b4nh/4eyDxRLVKZZPOw1+Or1d1EcRGjSwjo1rtDinHJ2GPyuDYw7MuajOFjY='} 2024-04-02T04:26:52.978 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634849', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-02 04:21:54.925808', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPH70r4gDZkVgtnf+XQEzpooFGq8G2FD8nASXwlDtS7Z7n55T3x+t3yEDXeqiB1V862PNVOszCjW6l+gUgJiLiE='} 2024-04-02T04:26:52.979 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-02T04:26:52.984 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-04-02T04:26:52.984 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-04-02T04:26:52.984 INFO:teuthology.run_tasks:Running task console_log... 2024-04-02T04:26:53.060 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb90f56ef70>, signals=[15]) 2024-04-02T04:26:53.061 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-02T04:26:53.066 INFO:teuthology.task.internal:Opening connections... 2024-04-02T04:26:53.066 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi053.front.sepia.ceph.com 2024-04-02T04:26:53.068 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T04:26:53.174 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi122.front.sepia.ceph.com 2024-04-02T04:26:53.175 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T04:26:53.265 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-02T04:26:53.270 DEBUG:teuthology.orchestra.run.smithi053:> uname -m 2024-04-02T04:26:53.288 INFO:teuthology.orchestra.run.smithi053.stdout:x86_64 2024-04-02T04:26:53.288 DEBUG:teuthology.orchestra.run.smithi053:> cat /etc/os-release 2024-04-02T04:26:53.345 INFO:teuthology.orchestra.run.smithi053.stdout:NAME="CentOS Stream" 2024-04-02T04:26:53.346 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION="9" 2024-04-02T04:26:53.346 INFO:teuthology.orchestra.run.smithi053.stdout:ID="centos" 2024-04-02T04:26:53.346 INFO:teuthology.orchestra.run.smithi053.stdout:ID_LIKE="rhel fedora" 2024-04-02T04:26:53.346 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION_ID="9" 2024-04-02T04:26:53.346 INFO:teuthology.orchestra.run.smithi053.stdout:PLATFORM_ID="platform:el9" 2024-04-02T04:26:53.346 INFO:teuthology.orchestra.run.smithi053.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-02T04:26:53.346 INFO:teuthology.orchestra.run.smithi053.stdout:ANSI_COLOR="0;31" 2024-04-02T04:26:53.346 INFO:teuthology.orchestra.run.smithi053.stdout:LOGO="fedora-logo-icon" 2024-04-02T04:26:53.346 INFO:teuthology.orchestra.run.smithi053.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-02T04:26:53.346 INFO:teuthology.orchestra.run.smithi053.stdout:HOME_URL="https://centos.org/" 2024-04-02T04:26:53.346 INFO:teuthology.orchestra.run.smithi053.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-02T04:26:53.346 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-02T04:26:53.346 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-02T04:26:53.347 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server 2024-04-02T04:26:53.370 DEBUG:teuthology.orchestra.run.smithi122:> uname -m 2024-04-02T04:26:53.386 INFO:teuthology.orchestra.run.smithi122.stdout:x86_64 2024-04-02T04:26:53.387 DEBUG:teuthology.orchestra.run.smithi122:> cat /etc/os-release 2024-04-02T04:26:53.442 INFO:teuthology.orchestra.run.smithi122.stdout:NAME="CentOS Stream" 2024-04-02T04:26:53.442 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION="9" 2024-04-02T04:26:53.442 INFO:teuthology.orchestra.run.smithi122.stdout:ID="centos" 2024-04-02T04:26:53.442 INFO:teuthology.orchestra.run.smithi122.stdout:ID_LIKE="rhel fedora" 2024-04-02T04:26:53.442 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_ID="9" 2024-04-02T04:26:53.442 INFO:teuthology.orchestra.run.smithi122.stdout:PLATFORM_ID="platform:el9" 2024-04-02T04:26:53.442 INFO:teuthology.orchestra.run.smithi122.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-02T04:26:53.443 INFO:teuthology.orchestra.run.smithi122.stdout:ANSI_COLOR="0;31" 2024-04-02T04:26:53.443 INFO:teuthology.orchestra.run.smithi122.stdout:LOGO="fedora-logo-icon" 2024-04-02T04:26:53.443 INFO:teuthology.orchestra.run.smithi122.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-02T04:26:53.443 INFO:teuthology.orchestra.run.smithi122.stdout:HOME_URL="https://centos.org/" 2024-04-02T04:26:53.443 INFO:teuthology.orchestra.run.smithi122.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-02T04:26:53.443 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-02T04:26:53.443 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-02T04:26:53.443 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2024-04-02T04:26:53.469 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-02T04:26:53.478 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-02T04:26:53.483 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-02T04:26:53.484 DEBUG:teuthology.orchestra.run.smithi053:> test '!' -e /home/ubuntu/cephtest 2024-04-02T04:26:53.490 DEBUG:teuthology.orchestra.run.smithi122:> test '!' -e /home/ubuntu/cephtest 2024-04-02T04:26:53.505 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-02T04:26:53.510 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-02T04:26:53.511 DEBUG:teuthology.orchestra.run.smithi053:> test -z $(ls -A /var/lib/ceph) 2024-04-02T04:26:53.544 DEBUG:teuthology.orchestra.run.smithi122:> test -z $(ls -A /var/lib/ceph) 2024-04-02T04:26:53.577 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-02T04:26:53.634 INFO:teuthology.run_tasks:Running task kernel... 2024-04-02T04:26:53.648 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-02T04:26:53.648 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-02T04:26:53.648 DEBUG:teuthology.orchestra.run.smithi053:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-02T04:26:53.649 DEBUG:teuthology.orchestra.run.smithi122:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-02T04:26:53.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:26:53.664 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2024-04-02T04:26:53.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:26:53.666 DEBUG:teuthology.orchestra.run.smithi053:> uname -r 2024-04-02T04:26:53.720 INFO:teuthology.orchestra.run.smithi122.stdout:5.14.0-361.el9.x86_64 2024-04-02T04:26:53.721 INFO:teuthology.task.kernel:Running kernel on smithi122: 5.14.0-361.el9.x86_64 2024-04-02T04:26:53.721 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum install -y kernel 2024-04-02T04:26:53.723 INFO:teuthology.orchestra.run.smithi053.stdout:5.14.0-361.el9.x86_64 2024-04-02T04:26:53.723 INFO:teuthology.task.kernel:Running kernel on smithi053: 5.14.0-361.el9.x86_64 2024-04-02T04:26:53.723 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum install -y kernel 2024-04-02T04:26:55.728 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-04-02T04:26:55.849 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-04-02T04:26:57.341 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-04-02T04:26:57.404 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-04-02T04:26:58.194 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-04-02T04:26:59.673 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.0 MB 00:01 2024-04-02T04:27:02.508 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-04-02T04:27:02.679 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - AppStream 5.5 MB/s | 19 MB 00:03 2024-04-02T04:27:06.691 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-04-02T04:27:07.171 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - CRB 6.3 MB/s | 6.1 MB 00:00 2024-04-02T04:27:08.409 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 16 kB 00:00 2024-04-02T04:27:08.572 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 16 kB 00:00 2024-04-02T04:27:10.478 INFO:teuthology.orchestra.run.smithi122.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-02T04:27:11.188 INFO:teuthology.orchestra.run.smithi053.stdout:Extra Packages for Enterprise Linux 7.9 MB/s | 21 MB 00:02 2024-04-02T04:27:15.607 INFO:teuthology.orchestra.run.smithi122.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-04-02T04:27:16.410 INFO:teuthology.orchestra.run.smithi053.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-02T04:27:17.956 INFO:teuthology.orchestra.run.smithi122.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-02T04:27:17.956 INFO:teuthology.orchestra.run.smithi122.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-02T04:27:18.050 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-02T04:27:18.055 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-02T04:27:18.055 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-04-02T04:27:18.055 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-02T04:27:18.055 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-02T04:27:18.055 INFO:teuthology.orchestra.run.smithi122.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-04-02T04:27:18.056 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-04-02T04:27:18.056 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-04-02T04:27:18.056 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-04-02T04:27:18.056 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-04-02T04:27:18.056 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-02T04:27:18.056 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-02T04:27:18.056 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-02T04:27:18.056 INFO:teuthology.orchestra.run.smithi122.stdout:Install 4 Packages 2024-04-02T04:27:18.056 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-02T04:27:18.057 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 76 M 2024-04-02T04:27:18.057 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 123 M 2024-04-02T04:27:18.057 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-02T04:27:18.209 INFO:teuthology.orchestra.run.smithi122.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-04-02T04:27:18.693 INFO:teuthology.orchestra.run.smithi122.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 24 MB/s | 15 MB 00:00 2024-04-02T04:27:18.776 INFO:teuthology.orchestra.run.smithi053.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-02T04:27:18.776 INFO:teuthology.orchestra.run.smithi053.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-02T04:27:18.865 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-04-02T04:27:18.882 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-04-02T04:27:18.882 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-04-02T04:27:18.882 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-04-02T04:27:18.882 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-04-02T04:27:18.882 INFO:teuthology.orchestra.run.smithi053.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-04-02T04:27:18.882 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-04-02T04:27:18.882 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-04-02T04:27:18.882 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-04-02T04:27:18.882 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-04-02T04:27:18.883 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-02T04:27:18.883 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-04-02T04:27:18.883 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-04-02T04:27:18.883 INFO:teuthology.orchestra.run.smithi053.stdout:Install 4 Packages 2024-04-02T04:27:18.883 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-02T04:27:18.883 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 76 M 2024-04-02T04:27:18.883 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 123 M 2024-04-02T04:27:18.883 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-04-02T04:27:19.053 INFO:teuthology.orchestra.run.smithi053.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-04-02T04:27:19.226 INFO:teuthology.orchestra.run.smithi122.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 29 MB/s | 34 MB 00:01 2024-04-02T04:27:19.637 INFO:teuthology.orchestra.run.smithi053.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 20 MB/s | 15 MB 00:00 2024-04-02T04:27:19.718 INFO:teuthology.orchestra.run.smithi122.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 18 MB/s | 28 MB 00:01 2024-04-02T04:27:19.721 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-02T04:27:19.721 INFO:teuthology.orchestra.run.smithi122.stdout:Total 46 MB/s | 76 MB 00:01 2024-04-02T04:27:19.722 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-02T04:27:20.095 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-02T04:27:20.096 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-02T04:27:20.254 INFO:teuthology.orchestra.run.smithi053.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 25 MB/s | 34 MB 00:01 2024-04-02T04:27:20.521 INFO:teuthology.orchestra.run.smithi053.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 19 MB/s | 28 MB 00:01 2024-04-02T04:27:20.524 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-04-02T04:27:20.524 INFO:teuthology.orchestra.run.smithi053.stdout:Total 47 MB/s | 76 MB 00:01 2024-04-02T04:27:20.524 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-04-02T04:27:20.936 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-02T04:27:20.937 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-02T04:27:20.958 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-04-02T04:27:20.958 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-04-02T04:27:21.826 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-04-02T04:27:21.826 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-04-02T04:27:23.321 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-02T04:27:23.988 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-04-02T04:27:24.071 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-02T04:27:24.262 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-04-02T04:27:24.860 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-04-02T04:27:24.946 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-02T04:27:26.826 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-02T04:27:26.978 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-02T04:27:27.827 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-02T04:27:27.958 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-02T04:27:31.674 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-02T04:27:31.710 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-04-02T04:27:33.025 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-02T04:27:33.099 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-04-02T04:27:36.445 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-04-02T04:27:38.194 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-04-02T04:28:05.867 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-04-02T04:28:05.867 INFO:teuthology.orchestra.run.smithi122.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-02T04:28:05.867 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-02T04:28:05.910 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-04-02T04:28:06.328 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-04-02T04:28:06.328 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-04-02T04:28:06.328 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-02T04:28:06.328 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-02T04:28:06.776 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-04-02T04:28:06.776 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-02T04:28:06.776 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-02T04:28:06.776 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-5.14.0-434.el9.x86_64 2024-04-02T04:28:06.776 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-04-02T04:28:06.776 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-04-02T04:28:06.776 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-04-02T04:28:06.776 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-02T04:28:06.776 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-02T04:28:07.105 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q kernel | sort -rV | head -n 1 2024-04-02T04:28:07.213 INFO:teuthology.orchestra.run.smithi122.stdout:kernel-5.14.0-434.el9.x86_64 2024-04-02T04:28:07.213 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-04-02T04:28:07.214 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-04-02T04:28:07.214 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-02T04:28:07.214 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-02T04:28:07.214 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-04-02T04:28:07.214 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi122.front.sepia.ceph.com, path=None, version=distro) 2024-04-02T04:28:07.214 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q kernel | sort -rV | head -n 1 2024-04-02T04:28:07.293 INFO:teuthology.orchestra.run.smithi122.stdout:kernel-5.14.0-434.el9.x86_64 2024-04-02T04:28:07.293 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-04-02T04:28:07.293 DEBUG:teuthology.orchestra.run.smithi122:> sudo rpm -qi grub2-tools 2024-04-02T04:28:07.327 INFO:teuthology.orchestra.run.smithi122.stdout:Name : grub2-tools 2024-04-02T04:28:07.328 INFO:teuthology.orchestra.run.smithi122.stdout:Epoch : 1 2024-04-02T04:28:07.328 INFO:teuthology.orchestra.run.smithi122.stdout:Version : 2.06 2024-04-02T04:28:07.328 INFO:teuthology.orchestra.run.smithi122.stdout:Release : 68.el9 2024-04-02T04:28:07.328 INFO:teuthology.orchestra.run.smithi122.stdout:Architecture: x86_64 2024-04-02T04:28:07.328 INFO:teuthology.orchestra.run.smithi122.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-02T04:28:07.328 INFO:teuthology.orchestra.run.smithi122.stdout:Group : Unspecified 2024-04-02T04:28:07.328 INFO:teuthology.orchestra.run.smithi122.stdout:Size : 8264254 2024-04-02T04:28:07.328 INFO:teuthology.orchestra.run.smithi122.stdout:License : GPLv3+ 2024-04-02T04:28:07.328 INFO:teuthology.orchestra.run.smithi122.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-02T04:28:07.328 INFO:teuthology.orchestra.run.smithi122.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-02T04:28:07.328 INFO:teuthology.orchestra.run.smithi122.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-02T04:28:07.328 INFO:teuthology.orchestra.run.smithi122.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-02T04:28:07.328 INFO:teuthology.orchestra.run.smithi122.stdout:Packager : builder@centos.org 2024-04-02T04:28:07.328 INFO:teuthology.orchestra.run.smithi122.stdout:Vendor : CentOS 2024-04-02T04:28:07.329 INFO:teuthology.orchestra.run.smithi122.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-02T04:28:07.329 INFO:teuthology.orchestra.run.smithi122.stdout:Summary : Support tools for GRUB. 2024-04-02T04:28:07.329 INFO:teuthology.orchestra.run.smithi122.stdout:Description : 2024-04-02T04:28:07.329 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-02T04:28:07.329 INFO:teuthology.orchestra.run.smithi122.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-02T04:28:07.329 INFO:teuthology.orchestra.run.smithi122.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-02T04:28:07.329 INFO:teuthology.orchestra.run.smithi122.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-02T04:28:07.329 INFO:teuthology.orchestra.run.smithi122.stdout:hardware devices. 2024-04-02T04:28:07.329 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-02T04:28:07.329 INFO:teuthology.orchestra.run.smithi122.stdout:This subpackage provides tools for support of all platforms. 2024-04-02T04:28:07.330 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-02T04:28:07.330 INFO:teuthology.task.kernel:Updating grub on smithi122 to boot 5.14.0-434.el9.x86_64 2024-04-02T04:28:07.330 DEBUG:teuthology.orchestra.run.smithi122:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-02T04:28:07.977 INFO:teuthology.orchestra.run.smithi122.stderr:Generating grub configuration file ... 2024-04-02T04:28:08.963 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-04-02T04:28:08.963 INFO:teuthology.orchestra.run.smithi053.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-02T04:28:08.963 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-02T04:28:09.007 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-04-02T04:28:09.333 INFO:teuthology.orchestra.run.smithi122.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-02T04:28:09.363 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-04-02T04:28:09.363 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-04-02T04:28:09.363 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-02T04:28:09.363 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-02T04:28:09.370 INFO:teuthology.orchestra.run.smithi122.stderr:done 2024-04-02T04:28:09.373 DEBUG:teuthology.orchestra.run.smithi122:> mktemp 2024-04-02T04:28:09.387 INFO:teuthology.orchestra.run.smithi122.stdout:/tmp/tmp.ry1vQGXuhF 2024-04-02T04:28:09.388 DEBUG:teuthology.orchestra.run.smithi122:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ry1vQGXuhF 2024-04-02T04:28:09.457 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0666 /tmp/tmp.ry1vQGXuhF 2024-04-02T04:28:09.602 DEBUG:teuthology.orchestra.remote:smithi122:/tmp/tmp.ry1vQGXuhF is 6KB 2024-04-02T04:28:09.622 DEBUG:teuthology.orchestra.run.smithi122:> rm -fr /tmp/tmp.ry1vQGXuhF 2024-04-02T04:28:09.636 DEBUG:teuthology.orchestra.run.smithi122:> sudo /bin/ls /boot/loader/entries || true 2024-04-02T04:28:09.707 INFO:teuthology.orchestra.run.smithi122.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-02T04:28:09.707 INFO:teuthology.orchestra.run.smithi122.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-02T04:28:09.707 INFO:teuthology.orchestra.run.smithi122.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-02T04:28:09.707 INFO:teuthology.orchestra.run.smithi122.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-04-02T04:28:09.708 DEBUG:teuthology.orchestra.run.smithi122:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-04-02T04:28:09.788 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-04-02T04:28:09.789 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-02T04:28:09.789 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-04-02T04:28:09.789 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-5.14.0-434.el9.x86_64 2024-04-02T04:28:09.789 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-04-02T04:28:09.789 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-04-02T04:28:09.789 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-04-02T04:28:09.789 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-02T04:28:09.789 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-04-02T04:28:09.865 DEBUG:teuthology.orchestra.run.smithi122:> sudo shutdown -r now 2024-04-02T04:28:10.114 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q kernel | sort -rV | head -n 1 2024-04-02T04:28:10.215 INFO:teuthology.orchestra.run.smithi053.stdout:kernel-5.14.0-434.el9.x86_64 2024-04-02T04:28:10.215 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-04-02T04:28:10.215 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-04-02T04:28:10.215 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-02T04:28:10.215 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-02T04:28:10.215 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-04-02T04:28:10.216 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi053.front.sepia.ceph.com, path=None, version=distro) 2024-04-02T04:28:10.216 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q kernel | sort -rV | head -n 1 2024-04-02T04:28:10.298 INFO:teuthology.orchestra.run.smithi053.stdout:kernel-5.14.0-434.el9.x86_64 2024-04-02T04:28:10.298 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-04-02T04:28:10.298 DEBUG:teuthology.orchestra.run.smithi053:> sudo rpm -qi grub2-tools 2024-04-02T04:28:10.345 INFO:teuthology.orchestra.run.smithi053.stdout:Name : grub2-tools 2024-04-02T04:28:10.345 INFO:teuthology.orchestra.run.smithi053.stdout:Epoch : 1 2024-04-02T04:28:10.346 INFO:teuthology.orchestra.run.smithi053.stdout:Version : 2.06 2024-04-02T04:28:10.346 INFO:teuthology.orchestra.run.smithi053.stdout:Release : 68.el9 2024-04-02T04:28:10.346 INFO:teuthology.orchestra.run.smithi053.stdout:Architecture: x86_64 2024-04-02T04:28:10.346 INFO:teuthology.orchestra.run.smithi053.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-02T04:28:10.346 INFO:teuthology.orchestra.run.smithi053.stdout:Group : Unspecified 2024-04-02T04:28:10.346 INFO:teuthology.orchestra.run.smithi053.stdout:Size : 8264254 2024-04-02T04:28:10.346 INFO:teuthology.orchestra.run.smithi053.stdout:License : GPLv3+ 2024-04-02T04:28:10.346 INFO:teuthology.orchestra.run.smithi053.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-02T04:28:10.346 INFO:teuthology.orchestra.run.smithi053.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-02T04:28:10.346 INFO:teuthology.orchestra.run.smithi053.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-02T04:28:10.346 INFO:teuthology.orchestra.run.smithi053.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-02T04:28:10.346 INFO:teuthology.orchestra.run.smithi053.stdout:Packager : builder@centos.org 2024-04-02T04:28:10.346 INFO:teuthology.orchestra.run.smithi053.stdout:Vendor : CentOS 2024-04-02T04:28:10.347 INFO:teuthology.orchestra.run.smithi053.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-02T04:28:10.347 INFO:teuthology.orchestra.run.smithi053.stdout:Summary : Support tools for GRUB. 2024-04-02T04:28:10.347 INFO:teuthology.orchestra.run.smithi053.stdout:Description : 2024-04-02T04:28:10.347 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-02T04:28:10.347 INFO:teuthology.orchestra.run.smithi053.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-02T04:28:10.347 INFO:teuthology.orchestra.run.smithi053.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-02T04:28:10.347 INFO:teuthology.orchestra.run.smithi053.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-02T04:28:10.347 INFO:teuthology.orchestra.run.smithi053.stdout:hardware devices. 2024-04-02T04:28:10.347 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-02T04:28:10.347 INFO:teuthology.orchestra.run.smithi053.stdout:This subpackage provides tools for support of all platforms. 2024-04-02T04:28:10.348 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-02T04:28:10.348 INFO:teuthology.task.kernel:Updating grub on smithi053 to boot 5.14.0-434.el9.x86_64 2024-04-02T04:28:10.348 DEBUG:teuthology.orchestra.run.smithi053:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-02T04:28:10.933 INFO:teuthology.orchestra.run.smithi053.stderr:Generating grub configuration file ... 2024-04-02T04:28:12.380 INFO:teuthology.orchestra.run.smithi053.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-02T04:28:12.400 INFO:teuthology.orchestra.run.smithi053.stderr:done 2024-04-02T04:28:12.402 DEBUG:teuthology.orchestra.run.smithi053:> mktemp 2024-04-02T04:28:12.417 INFO:teuthology.orchestra.run.smithi053.stdout:/tmp/tmp.OnB8Dd4nDr 2024-04-02T04:28:12.418 DEBUG:teuthology.orchestra.run.smithi053:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.OnB8Dd4nDr 2024-04-02T04:28:12.490 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0666 /tmp/tmp.OnB8Dd4nDr 2024-04-02T04:28:12.635 DEBUG:teuthology.orchestra.remote:smithi053:/tmp/tmp.OnB8Dd4nDr is 6KB 2024-04-02T04:28:12.684 DEBUG:teuthology.orchestra.run.smithi053:> rm -fr /tmp/tmp.OnB8Dd4nDr 2024-04-02T04:28:12.699 DEBUG:teuthology.orchestra.run.smithi053:> sudo /bin/ls /boot/loader/entries || true 2024-04-02T04:28:12.768 INFO:teuthology.orchestra.run.smithi053.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-02T04:28:12.768 INFO:teuthology.orchestra.run.smithi053.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-02T04:28:12.768 INFO:teuthology.orchestra.run.smithi053.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-02T04:28:12.768 INFO:teuthology.orchestra.run.smithi053.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-04-02T04:28:12.769 DEBUG:teuthology.orchestra.run.smithi053:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-04-02T04:28:12.931 DEBUG:teuthology.orchestra.run.smithi053:> sudo shutdown -r now 2024-04-02T04:28:39.894 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-04-02T04:28:39.895 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-02T04:28:39.896 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T04:28:42.938 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-04-02T04:28:42.938 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-04-02T04:28:42.938 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T04:28:58.275 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.122 2024-04-02T04:29:01.342 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-04-02T04:29:07.278 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-02T04:29:07.279 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T04:29:10.338 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.122 2024-04-02T04:29:10.344 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-04-02T04:29:10.344 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T04:29:22.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-02T04:29:22.352 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T04:29:28.734 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-04-02T04:29:40.746 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-04-02T04:29:40.747 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T04:29:40.766 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.122 2024-04-02T04:29:41.078 DEBUG:teuthology.orchestra.run.smithi053:> true 2024-04-02T04:29:41.583 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-04-02T04:29:41.583 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-434.el9.x86_64"... 2024-04-02T04:29:41.584 DEBUG:teuthology.orchestra.run.smithi053:> uname -r 2024-04-02T04:29:41.597 INFO:teuthology.orchestra.run.smithi053.stdout:5.14.0-434.el9.x86_64 2024-04-02T04:29:41.597 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-04-02T04:29:41.598 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-02T04:29:41.598 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-02T04:29:42.598 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-04-02T04:29:42.599 DEBUG:teuthology.orchestra.run.smithi053:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-02T04:29:42.693 INFO:teuthology.orchestra.run.smithi053.stdout:ttyS1 2024-04-02T04:29:42.724 DEBUG:teuthology.parallel:result is None 2024-04-02T04:29:55.767 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-02T04:29:55.768 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T04:29:56.136 DEBUG:teuthology.orchestra.run.smithi122:> true 2024-04-02T04:29:56.565 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-02T04:29:56.566 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-434.el9.x86_64"... 2024-04-02T04:29:56.566 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2024-04-02T04:29:56.580 INFO:teuthology.orchestra.run.smithi122.stdout:5.14.0-434.el9.x86_64 2024-04-02T04:29:56.580 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-04-02T04:29:56.580 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-02T04:29:56.580 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-02T04:29:57.581 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-04-02T04:29:57.582 DEBUG:teuthology.orchestra.run.smithi122:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-02T04:29:57.675 INFO:teuthology.orchestra.run.smithi122.stdout:ttyS1 2024-04-02T04:29:57.720 DEBUG:teuthology.parallel:result is None 2024-04-02T04:29:57.720 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-02T04:29:57.729 INFO:teuthology.task.internal:Creating test directory... 2024-04-02T04:29:57.730 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-02T04:29:57.733 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-02T04:29:57.759 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-02T04:29:57.833 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-02T04:29:57.843 INFO:teuthology.task.internal:Creating archive directory... 2024-04-02T04:29:57.844 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-02T04:29:57.847 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-02T04:29:57.888 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-02T04:29:57.895 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-02T04:29:57.896 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-02T04:29:57.908 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-02T04:29:57.939 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-02T04:29:57.953 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-02T04:29:57.964 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-02T04:29:57.977 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-02T04:29:57.979 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-02T04:29:57.986 INFO:teuthology.task.internal:Configuring sudo... 2024-04-02T04:29:57.986 DEBUG:teuthology.orchestra.run.smithi053:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-02T04:29:57.997 DEBUG:teuthology.orchestra.run.smithi122:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-02T04:29:58.049 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-02T04:29:58.062 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-02T04:29:58.062 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-02T04:29:58.067 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-02T04:29:58.107 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-02T04:29:58.164 DEBUG:teuthology.orchestra.run.smithi053:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-02T04:29:58.215 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-02T04:29:58.298 DEBUG:teuthology.orchestra.run.smithi053:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-02T04:29:58.342 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-02T04:29:58.342 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-02T04:29:58.410 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-02T04:29:58.454 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-02T04:29:58.512 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-02T04:29:58.597 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-02T04:29:58.640 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-02T04:29:58.640 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-02T04:29:58.707 DEBUG:teuthology.orchestra.run.smithi053:> sudo service rsyslog restart 2024-04-02T04:29:58.709 DEBUG:teuthology.orchestra.run.smithi122:> sudo service rsyslog restart 2024-04-02T04:29:58.760 INFO:teuthology.orchestra.run.smithi053.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-02T04:29:58.807 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-02T04:29:59.180 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-02T04:29:59.188 INFO:teuthology.task.internal:Starting timer... 2024-04-02T04:29:59.189 INFO:teuthology.run_tasks:Running task pcp... 2024-04-02T04:29:59.213 INFO:teuthology.run_tasks:Running task selinux... 2024-04-02T04:29:59.223 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-02T04:29:59.223 DEBUG:teuthology.orchestra.run.smithi053:> sudo service auditd rotate 2024-04-02T04:29:59.298 INFO:teuthology.orchestra.run.smithi053.stdout:Rotating logs: 2024-04-02T04:29:59.300 DEBUG:teuthology.orchestra.run.smithi122:> sudo service auditd rotate 2024-04-02T04:29:59.352 INFO:teuthology.orchestra.run.smithi122.stdout:Rotating logs: 2024-04-02T04:29:59.354 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-02T04:29:59.354 DEBUG:teuthology.orchestra.run.smithi053:> /usr/sbin/getenforce 2024-04-02T04:29:59.381 INFO:teuthology.orchestra.run.smithi053.stdout:Permissive 2024-04-02T04:29:59.381 DEBUG:teuthology.orchestra.run.smithi122:> /usr/sbin/getenforce 2024-04-02T04:29:59.405 INFO:teuthology.orchestra.run.smithi122.stdout:Permissive 2024-04-02T04:29:59.405 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi053.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi122.front.sepia.ceph.com': 'permissive'} 2024-04-02T04:29:59.405 DEBUG:teuthology.orchestra.run.smithi053:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-02T04:29:59.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:29:59.450 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-02T04:29:59.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T04:29:59.477 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-02T04:29:59.477 DEBUG:teuthology.orchestra.run.smithi053:> sudo /usr/sbin/setenforce permissive 2024-04-02T04:29:59.524 DEBUG:teuthology.orchestra.run.smithi122:> sudo /usr/sbin/setenforce permissive 2024-04-02T04:29:59.557 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-02T04:29:59.569 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-02T04:29:59.574 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-02T04:29:59.708 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-02T04:29:59.725 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-02T04:29:59.726 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi053.front.sepia.ceph.com,smithi122.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-02T04:34:24.616 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi053.front.sepia.ceph.com'), Remote(name='ubuntu@smithi122.front.sepia.ceph.com')] 2024-04-02T04:34:24.617 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-04-02T04:34:24.618 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T04:34:24.712 DEBUG:teuthology.orchestra.run.smithi053:> true 2024-04-02T04:34:24.793 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-04-02T04:34:24.794 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-02T04:34:24.795 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T04:34:24.882 DEBUG:teuthology.orchestra.run.smithi122:> true 2024-04-02T04:34:24.964 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-02T04:34:24.964 INFO:teuthology.run_tasks:Running task clock... 2024-04-02T04:34:24.974 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-02T04:34:24.975 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-02T04:34:24.975 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-02T04:34:24.978 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-02T04:34:24.978 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-02T04:34:25.008 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-02T04:34:25.025 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-02T04:34:25.034 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-02T04:34:25.051 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-02T04:34:25.056 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: ntpd: command not found 2024-04-02T04:34:25.070 INFO:teuthology.orchestra.run.smithi053.stdout:506 Cannot talk to daemon 2024-04-02T04:34:25.079 INFO:teuthology.orchestra.run.smithi122.stderr:sudo: ntpd: command not found 2024-04-02T04:34:25.087 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-02T04:34:25.093 INFO:teuthology.orchestra.run.smithi122.stdout:506 Cannot talk to daemon 2024-04-02T04:34:25.103 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-02T04:34:25.109 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-02T04:34:25.125 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-02T04:34:25.153 INFO:teuthology.orchestra.run.smithi053.stderr:bash: line 1: ntpq: command not found 2024-04-02T04:34:25.157 INFO:teuthology.orchestra.run.smithi053.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-02T04:34:25.157 INFO:teuthology.orchestra.run.smithi053.stdout:=============================================================================== 2024-04-02T04:34:25.157 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T04:34:25.157 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T04:34:25.157 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T04:34:25.157 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T04:34:25.181 INFO:teuthology.orchestra.run.smithi122.stderr:bash: line 1: ntpq: command not found 2024-04-02T04:34:25.184 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-02T04:34:25.185 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2024-04-02T04:34:25.185 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T04:34:25.185 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T04:34:25.185 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T04:34:25.185 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T04:34:25.186 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-02T04:34:25.289 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '6cec793d9073f0594e2364b9b687ecaf22424cca'} 2024-04-02T04:34:25.289 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-04-02T04:34:25.290 INFO:tasks.cephadm:Cluster fsid is 48e16742-f0aa-11ee-b647-cb9ed24678a4 2024-04-02T04:34:25.290 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-02T04:34:25.290 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-04-02T04:34:25.290 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi053': '172.21.15.53', 'mon.smithi122': '172.21.15.122'} 2024-04-02T04:34:25.290 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-02T04:34:25.291 DEBUG:teuthology.orchestra.run.smithi053:> sudo hostname $(hostname -s) 2024-04-02T04:34:25.319 DEBUG:teuthology.orchestra.run.smithi122:> sudo hostname $(hostname -s) 2024-04-02T04:34:25.346 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-04-02T04:34:25.347 DEBUG:teuthology.orchestra.run.smithi053:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-02T04:34:25.607 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 2 04:34 /home/ubuntu/cephtest/cephadm 2024-04-02T04:34:25.607 DEBUG:teuthology.orchestra.run.smithi122:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-02T04:34:25.850 INFO:teuthology.orchestra.run.smithi122.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 2 04:34 /home/ubuntu/cephtest/cephadm 2024-04-02T04:34:25.850 DEBUG:teuthology.orchestra.run.smithi053:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-02T04:34:25.867 DEBUG:teuthology.orchestra.run.smithi122:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-02T04:34:25.892 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-04-02T04:34:25.892 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-02T04:34:25.909 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-02T04:34:26.759 INFO:teuthology.orchestra.run.smithi053.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-02T04:34:26.760 INFO:teuthology.orchestra.run.smithi122.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-02T04:34:50.123 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:34:50.124 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-02T04:34:50.124 INFO:teuthology.orchestra.run.smithi053.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-02T04:34:50.124 INFO:teuthology.orchestra.run.smithi053.stdout: "repo_digests": [ 2024-04-02T04:34:50.124 INFO:teuthology.orchestra.run.smithi053.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-02T04:34:50.124 INFO:teuthology.orchestra.run.smithi053.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-02T04:34:50.124 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-04-02T04:34:50.124 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:34:50.192 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-04-02T04:34:50.192 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-02T04:34:50.192 INFO:teuthology.orchestra.run.smithi122.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-02T04:34:50.192 INFO:teuthology.orchestra.run.smithi122.stdout: "repo_digests": [ 2024-04-02T04:34:50.192 INFO:teuthology.orchestra.run.smithi122.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-02T04:34:50.192 INFO:teuthology.orchestra.run.smithi122.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-02T04:34:50.192 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-04-02T04:34:50.193 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-04-02T04:34:50.211 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /etc/ceph 2024-04-02T04:34:50.240 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /etc/ceph 2024-04-02T04:34:50.270 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 777 /etc/ceph 2024-04-02T04:34:50.307 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 777 /etc/ceph 2024-04-02T04:34:50.337 INFO:tasks.cephadm:Writing seed config... 2024-04-02T04:34:50.339 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-02T04:34:50.339 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-02T04:34:50.339 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-02T04:34:50.339 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-02T04:34:50.339 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-02T04:34:50.339 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-02T04:34:50.339 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-02T04:34:50.339 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-02T04:34:50.340 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-02T04:34:50.340 DEBUG:teuthology.orchestra.run.smithi053:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-02T04:34:50.361 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 48e16742-f0aa-11ee-b647-cb9ed24678a4 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-04-02T04:34:50.362 DEBUG:teuthology.orchestra.run.smithi053:mon.smithi053> sudo journalctl -f -n 0 -u ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@mon.smithi053.service 2024-04-02T04:34:50.404 INFO:tasks.cephadm:Bootstrapping... 2024-04-02T04:34:50.404 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.53 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-02T04:34:50.605 INFO:teuthology.orchestra.run.smithi053.stderr:-------------------------------------------------------------------------------- 2024-04-02T04:34:50.606 INFO:teuthology.orchestra.run.smithi053.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', '48e16742-f0aa-11ee-b647-cb9ed24678a4', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.53', '--skip-admin-label'] 2024-04-02T04:34:50.629 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: 4.9.4-dev 2024-04-02T04:34:50.631 INFO:teuthology.orchestra.run.smithi053.stderr:Verifying podman|docker is present... 2024-04-02T04:34:50.652 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: 4.9.4-dev 2024-04-02T04:34:50.653 INFO:teuthology.orchestra.run.smithi053.stderr:Verifying lvm2 is present... 2024-04-02T04:34:50.654 INFO:teuthology.orchestra.run.smithi053.stderr:Verifying time synchronization is in place... 2024-04-02T04:34:50.660 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-02T04:34:50.666 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: inactive 2024-04-02T04:34:50.673 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: enabled 2024-04-02T04:34:50.679 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: active 2024-04-02T04:34:50.679 INFO:teuthology.orchestra.run.smithi053.stderr:Unit chronyd.service is enabled and running 2024-04-02T04:34:50.679 INFO:teuthology.orchestra.run.smithi053.stderr:Repeating the final host check... 2024-04-02T04:34:50.701 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: 4.9.4-dev 2024-04-02T04:34:50.703 INFO:teuthology.orchestra.run.smithi053.stderr:podman (/bin/podman) version 4.9.4 is present 2024-04-02T04:34:50.703 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl is present 2024-04-02T04:34:50.703 INFO:teuthology.orchestra.run.smithi053.stderr:lvcreate is present 2024-04-02T04:34:50.709 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-02T04:34:50.715 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: inactive 2024-04-02T04:34:50.721 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: enabled 2024-04-02T04:34:50.727 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: active 2024-04-02T04:34:50.728 INFO:teuthology.orchestra.run.smithi053.stderr:Unit chronyd.service is enabled and running 2024-04-02T04:34:50.728 INFO:teuthology.orchestra.run.smithi053.stderr:Host looks OK 2024-04-02T04:34:50.728 INFO:teuthology.orchestra.run.smithi053.stderr:Cluster fsid: 48e16742-f0aa-11ee-b647-cb9ed24678a4 2024-04-02T04:34:50.728 INFO:teuthology.orchestra.run.smithi053.stderr:Acquiring lock 140395966807440 on /run/cephadm/48e16742-f0aa-11ee-b647-cb9ed24678a4.lock 2024-04-02T04:34:50.728 INFO:teuthology.orchestra.run.smithi053.stderr:Lock 140395966807440 acquired on /run/cephadm/48e16742-f0aa-11ee-b647-cb9ed24678a4.lock 2024-04-02T04:34:50.729 INFO:teuthology.orchestra.run.smithi053.stderr:Verifying IP 172.21.15.53 port 3300 ... 2024-04-02T04:34:50.729 INFO:teuthology.orchestra.run.smithi053.stderr:Verifying IP 172.21.15.53 port 6789 ... 2024-04-02T04:34:50.729 INFO:teuthology.orchestra.run.smithi053.stderr:Base mon IP is 172.21.15.53, final addrv is [v2:172.21.15.53:3300,v1:172.21.15.53:6789] 2024-04-02T04:34:50.731 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.53 metric 100 2024-04-02T04:34:50.731 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.53 metric 100 2024-04-02T04:34:50.734 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-04-02T04:34:50.734 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-02T04:34:50.734 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-02T04:34:50.736 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-02T04:34:50.736 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-04-02T04:34:50.736 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-02T04:34:50.736 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-02T04:34:50.737 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: inet6 fe80::ae1f:6bff:fef8:3718/64 scope link noprefixroute 2024-04-02T04:34:50.737 INFO:teuthology.orchestra.run.smithi053.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-02T04:34:50.737 INFO:teuthology.orchestra.run.smithi053.stderr:Mon IP `172.21.15.53` is in CIDR network `172.21.0.0/20` 2024-04-02T04:34:50.737 INFO:teuthology.orchestra.run.smithi053.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-02T04:34:50.738 INFO:teuthology.orchestra.run.smithi053.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-02T04:34:50.841 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-04-02T04:34:51.199 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: Getting image source signatures 2024-04-02T04:34:51.200 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-04-02T04:34:51.200 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-04-02T04:34:51.200 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-04-02T04:34:51.200 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-04-02T04:34:51.200 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-04-02T04:34:51.200 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-02T04:34:51.327 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: Writing manifest to image destination 2024-04-02T04:34:51.463 INFO:teuthology.orchestra.run.smithi053.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-02T04:34:51.852 INFO:teuthology.orchestra.run.smithi053.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-02T04:34:52.261 INFO:teuthology.orchestra.run.smithi053.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-02T04:34:52.261 INFO:teuthology.orchestra.run.smithi053.stderr:Extracting ceph user uid/gid from container image... 2024-04-02T04:34:52.678 INFO:teuthology.orchestra.run.smithi053.stderr:stat: 167 167 2024-04-02T04:34:53.142 INFO:teuthology.orchestra.run.smithi053.stderr:Creating initial keys... 2024-04-02T04:34:53.524 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-authtool: AQDtigtmwPwSGxAAU1Itfw9RItcDujpIJGQxlw== 2024-04-02T04:34:54.346 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-authtool: AQDuigtmqxRrEBAAbsx1qmzggi2z51gVPkVUCA== 2024-04-02T04:34:55.218 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph-authtool: AQDvigtmI8zGCRAAoL/MbwV1zY3z+Xi83aIW0Q== 2024-04-02T04:34:55.708 INFO:teuthology.orchestra.run.smithi053.stderr:Creating initial monmap... 2024-04-02T04:34:56.082 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-02T04:34:56.082 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-04-02T04:34:56.082 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 48e16742-f0aa-11ee-b647-cb9ed24678a4 2024-04-02T04:34:56.082 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-02T04:34:56.613 INFO:teuthology.orchestra.run.smithi053.stderr:monmaptool for smithi053 [v2:172.21.15.53:3300,v1:172.21.15.53:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-02T04:34:56.613 INFO:teuthology.orchestra.run.smithi053.stderr:setting min_mon_release = octopus 2024-04-02T04:34:56.613 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/monmaptool: set fsid to 48e16742-f0aa-11ee-b647-cb9ed24678a4 2024-04-02T04:34:56.613 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-02T04:34:56.613 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-04-02T04:34:56.613 INFO:teuthology.orchestra.run.smithi053.stderr:Creating mon... 2024-04-02T04:34:57.418 INFO:teuthology.orchestra.run.smithi053.stderr:create mon.smithi053 on 2024-04-02T04:34:57.620 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-02T04:34:57.770 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4.target → /etc/systemd/system/ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4.target. 2024-04-02T04:34:57.770 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4.target → /etc/systemd/system/ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4.target. 2024-04-02T04:34:58.060 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: Failed to reset failed state of unit ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@mon.smithi053.service: Unit ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@mon.smithi053.service not loaded. 2024-04-02T04:34:58.067 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: Created symlink /etc/systemd/system/ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4.target.wants/ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@mon.smithi053.service → /etc/systemd/system/ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@.service. 2024-04-02T04:34:58.702 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:34:58 smithi053 ceph-mon[24551]: mon.smithi053 is new leader, mons smithi053 in quorum (ranks 0) 2024-04-02T04:34:58.731 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: disabled 2024-04-02T04:34:58.738 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: inactive 2024-04-02T04:34:58.738 INFO:teuthology.orchestra.run.smithi053.stderr:firewalld.service is not enabled 2024-04-02T04:34:58.738 INFO:teuthology.orchestra.run.smithi053.stderr:Not possible to enable service . firewalld.service is not available 2024-04-02T04:34:58.739 INFO:teuthology.orchestra.run.smithi053.stderr:Waiting for mon to start... 2024-04-02T04:34:58.739 INFO:teuthology.orchestra.run.smithi053.stderr:Waiting for mon... 2024-04-02T04:34:58.952 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:34:58 smithi053 bash[24532]: 797e2f288934cefac6e95d2544595c571f0e4f5c6c53654ffda9714fe774f8b8 2024-04-02T04:34:58.952 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:34:58 smithi053 systemd[1]: Started Ceph mon.smithi053 for 48e16742-f0aa-11ee-b647-cb9ed24678a4. 2024-04-02T04:34:59.223 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: cluster: 2024-04-02T04:34:59.223 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: id: 48e16742-f0aa-11ee-b647-cb9ed24678a4 2024-04-02T04:34:59.223 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-04-02T04:34:59.223 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-04-02T04:34:59.223 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: services: 2024-04-02T04:34:59.223 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi053 (age 0.52379s) 2024-04-02T04:34:59.224 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: mgr: no daemons active 2024-04-02T04:34:59.224 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-04-02T04:34:59.224 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-04-02T04:34:59.224 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: data: 2024-04-02T04:34:59.224 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-04-02T04:34:59.224 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-04-02T04:34:59.224 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-04-02T04:34:59.224 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: pgs: 2024-04-02T04:34:59.224 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-04-02T04:34:59.758 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:34:59 smithi053 ceph-mon[24551]: mon.smithi053 is new leader, mons smithi053 in quorum (ranks 0) 2024-04-02T04:34:59.758 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:34:59 smithi053 ceph-mon[24551]: monmap e1: 1 mons at {smithi053=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0]} 2024-04-02T04:34:59.758 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:34:59 smithi053 ceph-mon[24551]: fsmap 2024-04-02T04:34:59.758 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:34:59 smithi053 ceph-mon[24551]: osdmap e1: 0 total, 0 up, 0 in 2024-04-02T04:34:59.758 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:34:59 smithi053 ceph-mon[24551]: mgrmap e1: no daemons active 2024-04-02T04:34:59.758 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:34:59 smithi053 ceph-mon[24551]: from='client.? 172.21.15.53:0/2507840960' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-02T04:34:59.761 INFO:teuthology.orchestra.run.smithi053.stderr:mon is available 2024-04-02T04:34:59.761 INFO:teuthology.orchestra.run.smithi053.stderr:Assimilating anything we can from ceph.conf... 2024-04-02T04:35:00.282 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-04-02T04:35:00.282 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: [global] 2024-04-02T04:35:00.282 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: fsid = 48e16742-f0aa-11ee-b647-cb9ed24678a4 2024-04-02T04:35:00.282 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.53:3300,v1:172.21.15.53:6789] 2024-04-02T04:35:00.282 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-02T04:35:00.283 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-02T04:35:00.283 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-02T04:35:00.283 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-02T04:35:00.283 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-04-02T04:35:00.283 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: [mgr] 2024-04-02T04:35:00.283 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-02T04:35:00.283 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-04-02T04:35:00.283 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: [osd] 2024-04-02T04:35:00.283 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-02T04:35:00.283 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-02T04:35:00.775 INFO:teuthology.orchestra.run.smithi053.stderr:Generating new minimal ceph.conf... 2024-04-02T04:35:00.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:00 smithi053 ceph-mon[24551]: from='client.? 172.21.15.53:0/2931716350' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-02T04:35:00.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:00 smithi053 ceph-mon[24551]: from='client.? 172.21.15.53:0/2931716350' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-02T04:35:01.681 INFO:teuthology.orchestra.run.smithi053.stderr:Restarting the monitor... 2024-04-02T04:35:01.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:01 smithi053 systemd[1]: Stopping Ceph mon.smithi053 for 48e16742-f0aa-11ee-b647-cb9ed24678a4... 2024-04-02T04:35:01.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:01 smithi053 ceph-mon[24551]: from='client.? 172.21.15.53:0/3919447569' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:35:01.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:01 smithi053 bash[24768]: Error: no container with name or ID "ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon.smithi053" found: no such container 2024-04-02T04:35:02.139 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:01 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi053[24547]: 2024-04-02T04:35:01.830+0000 7f5865fca700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi053 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T04:35:02.139 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:01 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi053[24547]: 2024-04-02T04:35:01.830+0000 7f5865fca700 -1 mon.smithi053@0(leader) e1 *** Got Signal Terminated *** 2024-04-02T04:35:02.139 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:01 smithi053 podman[24779]: 2024-04-02 04:35:01.872679546 +0000 UTC m=+0.143457557 container died 797e2f288934cefac6e95d2544595c571f0e4f5c6c53654ffda9714fe774f8b8 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi053, GIT_CLEAN=True, name=centos-stream, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, io.buildah.version=1.19.8, vendor=Red Hat, Inc., release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , version=8, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-02T04:35:02.139 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 podman[24779]: 2024-04-02 04:35:02.136010398 +0000 UTC m=+0.406788410 container cleanup 797e2f288934cefac6e95d2544595c571f0e4f5c6c53654ffda9714fe774f8b8 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi053, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, version=8, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, name=centos-stream, io.openshift.expose-services=, vcs-type=git, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream) 2024-04-02T04:35:02.400 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 bash[24779]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi053 2024-04-02T04:35:02.400 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 bash[24806]: Error: no container with name or ID "ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon.smithi053" found: no such container 2024-04-02T04:35:02.400 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 podman[24793]: 2024-04-02 04:35:02.378426105 +0000 UTC m=+0.499414463 container remove 797e2f288934cefac6e95d2544595c571f0e4f5c6c53654ffda9714fe774f8b8 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi053, io.buildah.version=1.19.8, GIT_CLEAN=True, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., architecture=x86_64, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, io.openshift.tags=base centos centos-stream, release=754, ceph=True, GIT_BRANCH=HEAD, io.openshift.expose-services=, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-02T04:35:02.401 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 systemd[1]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@mon.smithi053.service: Deactivated successfully. 2024-04-02T04:35:02.401 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 systemd[1]: Stopped Ceph mon.smithi053 for 48e16742-f0aa-11ee-b647-cb9ed24678a4. 2024-04-02T04:35:02.720 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 systemd[1]: Starting Ceph mon.smithi053 for 48e16742-f0aa-11ee-b647-cb9ed24678a4... 2024-04-02T04:35:02.720 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 podman[24871]: 2024-04-02 04:35:02.612477969 +0000 UTC m=+0.109755484 container create 665e4412f6da2eeedef9bdd05619b05b9dad0aaa37b331337a97a7b66b7684b0 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi053, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., RELEASE=HEAD, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, maintainer=Guillaume Abrioux , distribution-scope=public, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, name=centos-stream, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, release=754) 2024-04-02T04:35:02.720 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 podman[24871]: 2024-04-02 04:35:02.520669836 +0000 UTC m=+0.017947330 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-04-02T04:35:02.912 INFO:teuthology.orchestra.run.smithi053.stderr:Setting mon public_network to 172.21.0.0/20 2024-04-02T04:35:03.035 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 podman[24871]: 2024-04-02 04:35:02.831433206 +0000 UTC m=+0.328710695 container init 665e4412f6da2eeedef9bdd05619b05b9dad0aaa37b331337a97a7b66b7684b0 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi053, GIT_CLEAN=True, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, version=8, vendor=Red Hat, Inc., RELEASE=HEAD, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, io.k8s.display-name=CentOS Stream 8, vcs-type=git, GIT_BRANCH=HEAD, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , distribution-scope=public, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-02T04:35:03.035 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 podman[24871]: 2024-04-02 04:35:02.834372487 +0000 UTC m=+0.331649961 container start 665e4412f6da2eeedef9bdd05619b05b9dad0aaa37b331337a97a7b66b7684b0 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi053, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, release=754, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, io.buildah.version=1.19.8) 2024-04-02T04:35:03.035 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: set uid:gid to 167:167 (ceph:ceph) 2024-04-02T04:35:03.035 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-02T04:35:03.035 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: pidfile_write: ignore empty --pid-file 2024-04-02T04:35:03.035 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: load: jerasure load: lrc 2024-04-02T04:35:03.035 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: RocksDB version: 6.15.5 2024-04-02T04:35:03.036 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-02T04:35:03.036 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Compile date Apr 18 2022 2024-04-02T04:35:03.036 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: DB SUMMARY 2024-04-02T04:35:03.036 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: DB Session ID: C0DN0GKUCQ4FS39NFQV9 2024-04-02T04:35:03.036 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: CURRENT file: CURRENT 2024-04-02T04:35:03.036 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: IDENTITY file: IDENTITY 2024-04-02T04:35:03.036 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-04-02T04:35:03.036 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi053/store.db dir, Total Num: 1, files: 000008.sst 2024-04-02T04:35:03.036 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi053/store.db: 000010.log size: 82715 ; 2024-04-02T04:35:03.036 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.error_if_exists: 0 2024-04-02T04:35:03.036 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.create_if_missing: 0 2024-04-02T04:35:03.037 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.paranoid_checks: 1 2024-04-02T04:35:03.037 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-02T04:35:03.037 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.env: 0x563f148c6860 2024-04-02T04:35:03.037 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.fs: Posix File System 2024-04-02T04:35:03.037 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.info_log: 0x563f15a9ffa0 2024-04-02T04:35:03.037 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.max_file_opening_threads: 16 2024-04-02T04:35:03.037 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.statistics: (nil) 2024-04-02T04:35:03.037 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.use_fsync: 0 2024-04-02T04:35:03.037 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.max_log_file_size: 0 2024-04-02T04:35:03.037 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-02T04:35:03.037 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-02T04:35:03.037 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.keep_log_file_num: 1000 2024-04-02T04:35:03.037 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.recycle_log_file_num: 0 2024-04-02T04:35:03.037 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.allow_fallocate: 1 2024-04-02T04:35:03.038 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.allow_mmap_reads: 0 2024-04-02T04:35:03.038 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.allow_mmap_writes: 0 2024-04-02T04:35:03.038 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.use_direct_reads: 0 2024-04-02T04:35:03.038 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-02T04:35:03.038 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.create_missing_column_families: 0 2024-04-02T04:35:03.038 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.db_log_dir: 2024-04-02T04:35:03.038 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi053/store.db 2024-04-02T04:35:03.038 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-02T04:35:03.038 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-02T04:35:03.038 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-02T04:35:03.038 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-02T04:35:03.038 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-02T04:35:03.038 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-02T04:35:03.038 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.advise_random_on_open: 1 2024-04-02T04:35:03.039 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.db_write_buffer_size: 0 2024-04-02T04:35:03.039 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.write_buffer_manager: 0x563f15b90330 2024-04-02T04:35:03.039 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-02T04:35:03.039 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-02T04:35:03.039 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-02T04:35:03.039 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-02T04:35:03.039 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.rate_limiter: (nil) 2024-04-02T04:35:03.039 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-02T04:35:03.039 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.wal_recovery_mode: 2 2024-04-02T04:35:03.039 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.enable_thread_tracking: 0 2024-04-02T04:35:03.039 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.enable_pipelined_write: 0 2024-04-02T04:35:03.039 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.unordered_write: 0 2024-04-02T04:35:03.039 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-02T04:35:03.039 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-02T04:35:03.039 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-02T04:35:03.039 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-02T04:35:03.039 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.row_cache: None 2024-04-02T04:35:03.040 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.wal_filter: None 2024-04-02T04:35:03.040 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-02T04:35:03.040 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.allow_ingest_behind: 0 2024-04-02T04:35:03.040 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.preserve_deletes: 0 2024-04-02T04:35:03.040 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.two_write_queues: 0 2024-04-02T04:35:03.040 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.manual_wal_flush: 0 2024-04-02T04:35:03.040 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.atomic_flush: 0 2024-04-02T04:35:03.040 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-02T04:35:03.040 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-02T04:35:03.040 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-02T04:35:03.040 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.log_readahead_size: 0 2024-04-02T04:35:03.040 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-02T04:35:03.040 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.best_efforts_recovery: 0 2024-04-02T04:35:03.040 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-02T04:35:03.040 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-02T04:35:03.041 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.allow_data_in_errors: 0 2024-04-02T04:35:03.041 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.db_host_id: __hostname__ 2024-04-02T04:35:03.041 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.max_background_jobs: 2 2024-04-02T04:35:03.041 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.max_background_compactions: -1 2024-04-02T04:35:03.041 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.max_subcompactions: 1 2024-04-02T04:35:03.041 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-02T04:35:03.041 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-02T04:35:03.041 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-02T04:35:03.042 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.max_total_wal_size: 0 2024-04-02T04:35:03.042 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-02T04:35:03.042 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-02T04:35:03.042 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-02T04:35:03.042 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-02T04:35:03.042 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.max_open_files: -1 2024-04-02T04:35:03.042 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.bytes_per_sync: 0 2024-04-02T04:35:03.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-02T04:35:03.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-02T04:35:03.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.compaction_readahead_size: 0 2024-04-02T04:35:03.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.max_background_flushes: -1 2024-04-02T04:35:03.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Compression algorithms supported: 2024-04-02T04:35:03.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-02T04:35:03.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: kZSTD supported: 0 2024-04-02T04:35:03.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: kXpressCompression supported: 0 2024-04-02T04:35:03.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: kLZ4HCCompression supported: 1 2024-04-02T04:35:03.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: kLZ4Compression supported: 1 2024-04-02T04:35:03.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: kBZip2Compression supported: 0 2024-04-02T04:35:03.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: kZlibCompression supported: 1 2024-04-02T04:35:03.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: kSnappyCompression supported: 1 2024-04-02T04:35:03.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-02T04:35:03.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi053/store.db/MANIFEST-000009 2024-04-02T04:35:03.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-02T04:35:03.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-02T04:35:03.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.merge_operator: 2024-04-02T04:35:03.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.compaction_filter: None 2024-04-02T04:35:03.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.compaction_filter_factory: None 2024-04-02T04:35:03.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.sst_partitioner_factory: None 2024-04-02T04:35:03.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-02T04:35:03.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-02T04:35:03.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563f15a6bd00) 2024-04-02T04:35:03.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: cache_index_and_filter_blocks: 1 2024-04-02T04:35:03.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-02T04:35:03.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-02T04:35:03.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: pin_top_level_index_and_filter: 1 2024-04-02T04:35:03.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: index_type: 0 2024-04-02T04:35:03.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: data_block_index_type: 0 2024-04-02T04:35:03.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: index_shortening: 1 2024-04-02T04:35:03.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-02T04:35:03.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: hash_index_allow_collision: 1 2024-04-02T04:35:03.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: checksum: 1 2024-04-02T04:35:03.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: no_block_cache: 0 2024-04-02T04:35:03.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: block_cache: 0x563f15ad6170 2024-04-02T04:35:03.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: block_cache_name: BinnedLRUCache 2024-04-02T04:35:03.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: block_cache_options: 2024-04-02T04:35:03.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: capacity : 536870912 2024-04-02T04:35:03.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: num_shard_bits : 4 2024-04-02T04:35:03.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: strict_capacity_limit : 0 2024-04-02T04:35:03.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: high_pri_pool_ratio: 0.000 2024-04-02T04:35:03.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: block_cache_compressed: (nil) 2024-04-02T04:35:03.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: persistent_cache: (nil) 2024-04-02T04:35:03.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: block_size: 4096 2024-04-02T04:35:03.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: block_size_deviation: 10 2024-04-02T04:35:03.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: block_restart_interval: 16 2024-04-02T04:35:03.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: index_block_restart_interval: 1 2024-04-02T04:35:03.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: metadata_block_size: 4096 2024-04-02T04:35:03.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: partition_filters: 0 2024-04-02T04:35:03.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: use_delta_encoding: 1 2024-04-02T04:35:03.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-02T04:35:03.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: whole_key_filtering: 1 2024-04-02T04:35:03.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: verify_compression: 0 2024-04-02T04:35:03.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: read_amp_bytes_per_bit: 0 2024-04-02T04:35:03.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: format_version: 4 2024-04-02T04:35:03.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: enable_index_compression: 1 2024-04-02T04:35:03.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: block_align: 0 2024-04-02T04:35:03.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.write_buffer_size: 33554432 2024-04-02T04:35:03.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.max_write_buffer_number: 2 2024-04-02T04:35:03.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.compression: NoCompression 2024-04-02T04:35:03.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.bottommost_compression: Disabled 2024-04-02T04:35:03.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.prefix_extractor: nullptr 2024-04-02T04:35:03.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-02T04:35:03.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.num_levels: 7 2024-04-02T04:35:03.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-02T04:35:03.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-02T04:35:03.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-02T04:35:03.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-02T04:35:03.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-02T04:35:03.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-02T04:35:03.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-02T04:35:03.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-02T04:35:03.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-02T04:35:03.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-02T04:35:03.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-02T04:35:03.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.compression_opts.level: 32767 2024-04-02T04:35:03.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.compression_opts.strategy: 0 2024-04-02T04:35:03.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-02T04:35:03.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-02T04:35:03.048 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-02T04:35:03.048 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.compression_opts.enabled: false 2024-04-02T04:35:03.048 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-02T04:35:03.048 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-02T04:35:03.048 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-02T04:35:03.048 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.target_file_size_base: 67108864 2024-04-02T04:35:03.048 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-02T04:35:03.048 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-02T04:35:03.048 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-02T04:35:03.048 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-02T04:35:03.048 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-02T04:35:03.048 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-02T04:35:03.048 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-02T04:35:03.048 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-02T04:35:03.048 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-02T04:35:03.048 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-02T04:35:03.049 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-02T04:35:03.049 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-02T04:35:03.049 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-02T04:35:03.049 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.arena_block_size: 4194304 2024-04-02T04:35:03.049 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-02T04:35:03.049 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-02T04:35:03.049 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-02T04:35:03.049 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.disable_auto_compactions: 0 2024-04-02T04:35:03.049 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-02T04:35:03.049 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-02T04:35:03.049 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-02T04:35:03.049 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-02T04:35:03.049 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-02T04:35:03.049 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-02T04:35:03.049 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-02T04:35:03.049 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-02T04:35:03.049 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-02T04:35:03.050 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-02T04:35:03.050 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.table_properties_collectors: 2024-04-02T04:35:03.050 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.inplace_update_support: 0 2024-04-02T04:35:03.050 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-02T04:35:03.050 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-02T04:35:03.050 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-02T04:35:03.050 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-02T04:35:03.050 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.bloom_locality: 0 2024-04-02T04:35:03.050 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.max_successive_merges: 0 2024-04-02T04:35:03.050 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-02T04:35:03.050 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.paranoid_file_checks: 0 2024-04-02T04:35:03.050 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.force_consistency_checks: 1 2024-04-02T04:35:03.050 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.report_bg_io_stats: 0 2024-04-02T04:35:03.050 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.ttl: 2592000 2024-04-02T04:35:03.051 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-02T04:35:03.051 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.enable_blob_files: false 2024-04-02T04:35:03.051 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.min_blob_size: 0 2024-04-02T04:35:03.051 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.blob_file_size: 268435456 2024-04-02T04:35:03.051 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-02T04:35:03.051 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-02T04:35:03.051 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-02T04:35:03.051 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi053/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-02T04:35:03.051 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-04-02T04:35:03.051 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-04-02T04:35:03.051 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712032502860444, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-04-02T04:35:03.051 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-04-02T04:35:03.051 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-02T04:35:03.051 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712032502861174, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79397, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77661, "index_size": 224, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 9884, "raw_average_key_size": 47, "raw_value_size": 71970, "raw_average_value_size": 344, "num_data_blocks": 10, "num_entries": 209, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1712032502, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "9c22fed5-2c67-4d91-9b93-3b43a6c7a70b", "db_session_id": "C0DN0GKUCQ4FS39NFQV9"}} 2024-04-02T04:35:03.051 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-04-02T04:35:03.051 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712032502861519, "job": 1, "event": "recovery_finished"} 2024-04-02T04:35:03.052 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi053/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-02T04:35:03.052 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x563f15abca80 2024-04-02T04:35:03.052 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: DB pointer 0x563f15acc000 2024-04-02T04:35:03.052 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-02T04:35:03.052 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-02T04:35:03.052 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: ** DB Stats ** 2024-04-02T04:35:03.052 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-02T04:35:03.052 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-02T04:35:03.052 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-02T04:35:03.052 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-02T04:35:03.052 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-02T04:35:03.052 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-02T04:35:03.052 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-02T04:35:03.052 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: 2024-04-02T04:35:03.052 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: ** Compaction Stats [default] ** 2024-04-02T04:35:03.052 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-02T04:35:03.052 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-02T04:35:03.053 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: L0 2/0 79.33 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.4 0.00 0.00 1 0.000 0 0 2024-04-02T04:35:03.053 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Sum 2/0 79.33 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.4 0.00 0.00 1 0.000 0 0 2024-04-02T04:35:03.053 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.4 0.00 0.00 1 0.000 0 0 2024-04-02T04:35:03.053 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: 2024-04-02T04:35:03.053 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: ** Compaction Stats [default] ** 2024-04-02T04:35:03.053 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-02T04:35:03.053 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-02T04:35:03.053 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 151.4 0.00 0.00 1 0.000 0 0 2024-04-02T04:35:03.053 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-02T04:35:03.053 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-02T04:35:03.053 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-02T04:35:03.053 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-02T04:35:03.053 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-02T04:35:03.053 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-02T04:35:03.053 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Cumulative compaction: 0.00 GB write, 25.16 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-02T04:35:03.054 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Interval compaction: 0.00 GB write, 25.16 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-02T04:35:03.054 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-02T04:35:03.054 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: 2024-04-02T04:35:03.054 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-02T04:35:03.054 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: 2024-04-02T04:35:03.054 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: ** Compaction Stats [default] ** 2024-04-02T04:35:03.054 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-02T04:35:03.054 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-02T04:35:03.054 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: L0 2/0 79.33 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.4 0.00 0.00 1 0.000 0 0 2024-04-02T04:35:03.054 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Sum 2/0 79.33 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.4 0.00 0.00 1 0.000 0 0 2024-04-02T04:35:03.054 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-02T04:35:03.054 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: 2024-04-02T04:35:03.054 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: ** Compaction Stats [default] ** 2024-04-02T04:35:03.054 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-02T04:35:03.055 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-02T04:35:03.055 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 151.4 0.00 0.00 1 0.000 0 0 2024-04-02T04:35:03.055 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-02T04:35:03.055 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-02T04:35:03.055 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-02T04:35:03.055 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-02T04:35:03.055 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-02T04:35:03.055 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-02T04:35:03.055 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Cumulative compaction: 0.00 GB write, 24.88 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-02T04:35:03.055 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-02T04:35:03.055 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-02T04:35:03.055 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: 2024-04-02T04:35:03.055 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-02T04:35:03.055 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: starting mon.smithi053 rank 0 at public addrs [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] at bind addrs [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi053 fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 2024-04-02T04:35:03.055 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: mon.smithi053@-1(???) e1 preinit fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 2024-04-02T04:35:03.055 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: mon.smithi053@-1(???).mds e1 new map 2024-04-02T04:35:03.056 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: mon.smithi053@-1(???).mds e1 print_map 2024-04-02T04:35:03.056 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: e1 2024-04-02T04:35:03.056 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-02T04:35:03.056 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-02T04:35:03.056 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: legacy client fscid: -1 2024-04-02T04:35:03.056 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: 2024-04-02T04:35:03.056 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: No filesystems configured 2024-04-02T04:35:03.056 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: mon.smithi053@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-02T04:35:03.056 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: mon.smithi053@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-02T04:35:03.056 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: mon.smithi053@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-02T04:35:03.056 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: mon.smithi053@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-02T04:35:03.056 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: mon.smithi053@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-02T04:35:03.056 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 bash[24871]: 665e4412f6da2eeedef9bdd05619b05b9dad0aaa37b331337a97a7b66b7684b0 2024-04-02T04:35:03.056 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 systemd[1]: Started Ceph mon.smithi053 for 48e16742-f0aa-11ee-b647-cb9ed24678a4. 2024-04-02T04:35:03.056 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: mon.smithi053 is new leader, mons smithi053 in quorum (ranks 0) 2024-04-02T04:35:03.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: monmap e1: 1 mons at {smithi053=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0]} 2024-04-02T04:35:03.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: fsmap 2024-04-02T04:35:03.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: osdmap e1: 0 total, 0 up, 0 in 2024-04-02T04:35:03.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:02 smithi053 ceph-mon[24889]: mgrmap e1: no daemons active 2024-04-02T04:35:03.935 INFO:teuthology.orchestra.run.smithi053.stderr:Wrote config to /etc/ceph/ceph.conf 2024-04-02T04:35:03.935 INFO:teuthology.orchestra.run.smithi053.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-02T04:35:03.935 INFO:teuthology.orchestra.run.smithi053.stderr:Creating mgr... 2024-04-02T04:35:03.936 INFO:teuthology.orchestra.run.smithi053.stderr:Verifying port 9283 ... 2024-04-02T04:35:04.112 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: Failed to reset failed state of unit ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@mgr.smithi053.legdmv.service: Unit ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@mgr.smithi053.legdmv.service not loaded. 2024-04-02T04:35:04.118 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: Created symlink /etc/systemd/system/ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4.target.wants/ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@mgr.smithi053.legdmv.service → /etc/systemd/system/ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@.service. 2024-04-02T04:35:04.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:04 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/1281925207' entity='client.admin' 2024-04-02T04:35:04.823 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: disabled 2024-04-02T04:35:04.830 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: inactive 2024-04-02T04:35:04.830 INFO:teuthology.orchestra.run.smithi053.stderr:firewalld.service is not enabled 2024-04-02T04:35:04.831 INFO:teuthology.orchestra.run.smithi053.stderr:Not possible to enable service . firewalld.service is not available 2024-04-02T04:35:04.839 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: disabled 2024-04-02T04:35:04.846 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: inactive 2024-04-02T04:35:04.847 INFO:teuthology.orchestra.run.smithi053.stderr:firewalld.service is not enabled 2024-04-02T04:35:04.847 INFO:teuthology.orchestra.run.smithi053.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-04-02T04:35:04.847 INFO:teuthology.orchestra.run.smithi053.stderr:Waiting for mgr to start... 2024-04-02T04:35:04.847 INFO:teuthology.orchestra.run.smithi053.stderr:Waiting for mgr... 2024-04-02T04:35:05.341 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-04-02T04:35:05.342 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: { 2024-04-02T04:35:05.342 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "fsid": "48e16742-f0aa-11ee-b647-cb9ed24678a4", 2024-04-02T04:35:05.342 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "health": { 2024-04-02T04:35:05.342 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-02T04:35:05.342 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "checks": {}, 2024-04-02T04:35:05.342 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "mutes": [] 2024-04-02T04:35:05.342 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-04-02T04:35:05.342 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-02T04:35:05.342 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum": [ 2024-04-02T04:35:05.342 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 0 2024-04-02T04:35:05.342 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-04-02T04:35:05.342 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-02T04:35:05.343 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "smithi053" 2024-04-02T04:35:05.343 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-04-02T04:35:05.343 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-04-02T04:35:05.343 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "monmap": { 2024-04-02T04:35:05.343 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-02T04:35:05.343 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-02T04:35:05.343 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-02T04:35:05.343 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-04-02T04:35:05.343 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osdmap": { 2024-04-02T04:35:05.343 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-02T04:35:05.343 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-02T04:35:05.343 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-02T04:35:05.343 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-02T04:35:05.343 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-02T04:35:05.343 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-02T04:35:05.344 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-02T04:35:05.344 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-04-02T04:35:05.344 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "pgmap": { 2024-04-02T04:35:05.344 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-02T04:35:05.344 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-02T04:35:05.344 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-02T04:35:05.344 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-02T04:35:05.344 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-02T04:35:05.344 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-02T04:35:05.344 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-02T04:35:05.344 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-02T04:35:05.344 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-04-02T04:35:05.344 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "fsmap": { 2024-04-02T04:35:05.344 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-02T04:35:05.345 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-02T04:35:05.345 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-02T04:35:05.345 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-04-02T04:35:05.345 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-02T04:35:05.345 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "available": false, 2024-04-02T04:35:05.345 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-02T04:35:05.345 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "modules": [ 2024-04-02T04:35:05.345 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "iostat", 2024-04-02T04:35:05.345 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "nfs", 2024-04-02T04:35:05.345 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "restful" 2024-04-02T04:35:05.345 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-04-02T04:35:05.345 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "services": {} 2024-04-02T04:35:05.345 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-04-02T04:35:05.345 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "servicemap": { 2024-04-02T04:35:05.345 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-02T04:35:05.346 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "modified": "2024-04-02T04:34:58.695632+0000", 2024-04-02T04:35:05.346 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "services": {} 2024-04-02T04:35:05.346 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-04-02T04:35:05.346 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-02T04:35:05.346 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: } 2024-04-02T04:35:05.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:05 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/2065890579' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-02T04:35:05.968 INFO:teuthology.orchestra.run.smithi053.stderr:mgr not available, waiting (1/15)... 2024-04-02T04:35:08.495 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-04-02T04:35:08.495 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: { 2024-04-02T04:35:08.495 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "fsid": "48e16742-f0aa-11ee-b647-cb9ed24678a4", 2024-04-02T04:35:08.496 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "health": { 2024-04-02T04:35:08.496 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-02T04:35:08.496 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "checks": {}, 2024-04-02T04:35:08.496 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "mutes": [] 2024-04-02T04:35:08.496 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-04-02T04:35:08.496 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-02T04:35:08.496 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum": [ 2024-04-02T04:35:08.496 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 0 2024-04-02T04:35:08.496 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-04-02T04:35:08.496 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-02T04:35:08.496 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "smithi053" 2024-04-02T04:35:08.496 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-04-02T04:35:08.496 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-04-02T04:35:08.496 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "monmap": { 2024-04-02T04:35:08.496 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-02T04:35:08.497 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-02T04:35:08.497 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-02T04:35:08.497 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-04-02T04:35:08.497 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osdmap": { 2024-04-02T04:35:08.497 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-02T04:35:08.497 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-02T04:35:08.497 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-02T04:35:08.497 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-02T04:35:08.497 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-02T04:35:08.500 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-02T04:35:08.501 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-02T04:35:08.501 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-04-02T04:35:08.501 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "pgmap": { 2024-04-02T04:35:08.501 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-02T04:35:08.501 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-02T04:35:08.501 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-02T04:35:08.501 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-02T04:35:08.501 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-02T04:35:08.501 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-02T04:35:08.501 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-02T04:35:08.501 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-02T04:35:08.501 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-04-02T04:35:08.501 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "fsmap": { 2024-04-02T04:35:08.501 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-02T04:35:08.501 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-02T04:35:08.501 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-02T04:35:08.502 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-04-02T04:35:08.502 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-02T04:35:08.502 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "available": false, 2024-04-02T04:35:08.502 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-02T04:35:08.502 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "modules": [ 2024-04-02T04:35:08.502 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "iostat", 2024-04-02T04:35:08.502 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "nfs", 2024-04-02T04:35:08.502 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "restful" 2024-04-02T04:35:08.502 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-04-02T04:35:08.502 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "services": {} 2024-04-02T04:35:08.502 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-04-02T04:35:08.502 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "servicemap": { 2024-04-02T04:35:08.502 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-02T04:35:08.503 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "modified": "2024-04-02T04:34:58.695632+0000", 2024-04-02T04:35:08.503 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "services": {} 2024-04-02T04:35:08.503 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-04-02T04:35:08.503 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-02T04:35:08.503 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: } 2024-04-02T04:35:09.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:08 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/2059116489' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-02T04:35:09.172 INFO:teuthology.orchestra.run.smithi053.stderr:mgr not available, waiting (2/15)... 2024-04-02T04:35:11.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:10 smithi053 ceph-mon[24889]: Activating manager daemon smithi053.legdmv 2024-04-02T04:35:11.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:10 smithi053 ceph-mon[24889]: mgrmap e2: smithi053.legdmv(active, starting, since 0.00259563s) 2024-04-02T04:35:11.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:10 smithi053 ceph-mon[24889]: from='mgr.14100 172.21.15.53:0/2595882845' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-02T04:35:11.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:10 smithi053 ceph-mon[24889]: from='mgr.14100 172.21.15.53:0/2595882845' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-02T04:35:11.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:10 smithi053 ceph-mon[24889]: from='mgr.14100 172.21.15.53:0/2595882845' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-02T04:35:11.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:10 smithi053 ceph-mon[24889]: from='mgr.14100 172.21.15.53:0/2595882845' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi053"}]: dispatch 2024-04-02T04:35:11.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:10 smithi053 ceph-mon[24889]: from='mgr.14100 172.21.15.53:0/2595882845' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mgr metadata", "who": "smithi053.legdmv", "id": "smithi053.legdmv"}]: dispatch 2024-04-02T04:35:11.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:10 smithi053 ceph-mon[24889]: Manager daemon smithi053.legdmv is now available 2024-04-02T04:35:11.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:10 smithi053 ceph-mon[24889]: from='mgr.14100 172.21.15.53:0/2595882845' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:35:11.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:10 smithi053 ceph-mon[24889]: from='mgr.14100 172.21.15.53:0/2595882845' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/trash_purge_schedule"}]: dispatch 2024-04-02T04:35:11.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:10 smithi053 ceph-mon[24889]: from='mgr.14100 172.21.15.53:0/2595882845' entity='mgr.smithi053.legdmv' 2024-04-02T04:35:11.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:10 smithi053 ceph-mon[24889]: from='mgr.14100 172.21.15.53:0/2595882845' entity='mgr.smithi053.legdmv' 2024-04-02T04:35:11.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:10 smithi053 ceph-mon[24889]: from='mgr.14100 172.21.15.53:0/2595882845' entity='mgr.smithi053.legdmv' 2024-04-02T04:35:11.653 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-04-02T04:35:11.653 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: { 2024-04-02T04:35:11.653 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "fsid": "48e16742-f0aa-11ee-b647-cb9ed24678a4", 2024-04-02T04:35:11.653 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "health": { 2024-04-02T04:35:11.653 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-02T04:35:11.653 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "checks": {}, 2024-04-02T04:35:11.653 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "mutes": [] 2024-04-02T04:35:11.653 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-04-02T04:35:11.653 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-02T04:35:11.654 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum": [ 2024-04-02T04:35:11.654 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 0 2024-04-02T04:35:11.654 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-04-02T04:35:11.654 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-02T04:35:11.654 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "smithi053" 2024-04-02T04:35:11.654 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-04-02T04:35:11.654 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum_age": 8, 2024-04-02T04:35:11.654 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "monmap": { 2024-04-02T04:35:11.654 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-02T04:35:11.654 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-02T04:35:11.654 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-02T04:35:11.654 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-04-02T04:35:11.654 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osdmap": { 2024-04-02T04:35:11.654 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-02T04:35:11.655 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-02T04:35:11.655 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-02T04:35:11.655 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-02T04:35:11.655 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-02T04:35:11.655 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-02T04:35:11.655 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-02T04:35:11.655 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-04-02T04:35:11.655 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "pgmap": { 2024-04-02T04:35:11.655 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-02T04:35:11.655 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-02T04:35:11.655 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-02T04:35:11.655 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-02T04:35:11.655 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-02T04:35:11.655 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-02T04:35:11.655 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-02T04:35:11.656 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-02T04:35:11.656 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-04-02T04:35:11.656 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "fsmap": { 2024-04-02T04:35:11.656 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-02T04:35:11.656 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-02T04:35:11.656 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-02T04:35:11.656 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-04-02T04:35:11.656 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-02T04:35:11.656 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "available": false, 2024-04-02T04:35:11.656 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-02T04:35:11.656 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "modules": [ 2024-04-02T04:35:11.656 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "iostat", 2024-04-02T04:35:11.656 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "nfs", 2024-04-02T04:35:11.656 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "restful" 2024-04-02T04:35:11.656 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-04-02T04:35:11.657 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "services": {} 2024-04-02T04:35:11.657 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-04-02T04:35:11.657 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "servicemap": { 2024-04-02T04:35:11.657 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-02T04:35:11.657 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "modified": "2024-04-02T04:34:58.695632+0000", 2024-04-02T04:35:11.657 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "services": {} 2024-04-02T04:35:11.657 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-04-02T04:35:11.657 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-02T04:35:11.657 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: } 2024-04-02T04:35:12.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:11 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/2863521127' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-02T04:35:12.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:11 smithi053 ceph-mon[24889]: mgrmap e3: smithi053.legdmv(active, since 1.0055s) 2024-04-02T04:35:13.620 INFO:teuthology.orchestra.run.smithi053.stderr:mgr not available, waiting (3/15)... 2024-04-02T04:35:14.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:13 smithi053 ceph-mon[24889]: mgrmap e4: smithi053.legdmv(active, since 2s) 2024-04-02T04:35:19.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:19 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/4090675491' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-02T04:35:19.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-04-02T04:35:19.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: { 2024-04-02T04:35:19.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "fsid": "48e16742-f0aa-11ee-b647-cb9ed24678a4", 2024-04-02T04:35:19.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "health": { 2024-04-02T04:35:19.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-02T04:35:19.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "checks": {}, 2024-04-02T04:35:19.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "mutes": [] 2024-04-02T04:35:19.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-04-02T04:35:19.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-02T04:35:19.597 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum": [ 2024-04-02T04:35:19.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 0 2024-04-02T04:35:19.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-04-02T04:35:19.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-02T04:35:19.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "smithi053" 2024-04-02T04:35:19.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-04-02T04:35:19.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "quorum_age": 16, 2024-04-02T04:35:19.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "monmap": { 2024-04-02T04:35:19.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-02T04:35:19.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-02T04:35:19.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-02T04:35:19.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-04-02T04:35:19.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osdmap": { 2024-04-02T04:35:19.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-02T04:35:19.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-02T04:35:19.598 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-02T04:35:19.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-02T04:35:19.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-02T04:35:19.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-02T04:35:19.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-02T04:35:19.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-04-02T04:35:19.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "pgmap": { 2024-04-02T04:35:19.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-02T04:35:19.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-02T04:35:19.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-02T04:35:19.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-02T04:35:19.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-02T04:35:19.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-02T04:35:19.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-02T04:35:19.599 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-02T04:35:19.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-04-02T04:35:19.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "fsmap": { 2024-04-02T04:35:19.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-02T04:35:19.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-02T04:35:19.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-02T04:35:19.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-04-02T04:35:19.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-02T04:35:19.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "available": true, 2024-04-02T04:35:19.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-02T04:35:19.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "modules": [ 2024-04-02T04:35:19.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "iostat", 2024-04-02T04:35:19.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "nfs", 2024-04-02T04:35:19.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "restful" 2024-04-02T04:35:19.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ], 2024-04-02T04:35:19.600 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "services": {} 2024-04-02T04:35:19.601 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-04-02T04:35:19.601 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "servicemap": { 2024-04-02T04:35:19.601 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-02T04:35:19.601 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "modified": "2024-04-02T04:34:58.695632+0000", 2024-04-02T04:35:19.601 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "services": {} 2024-04-02T04:35:19.601 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: }, 2024-04-02T04:35:19.601 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-02T04:35:19.601 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: } 2024-04-02T04:35:24.119 INFO:teuthology.orchestra.run.smithi053.stderr:mgr is available 2024-04-02T04:35:24.746 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-04-02T04:35:24.746 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: [global] 2024-04-02T04:35:24.746 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: fsid = 48e16742-f0aa-11ee-b647-cb9ed24678a4 2024-04-02T04:35:24.746 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-02T04:35:24.746 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-02T04:35:24.746 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-02T04:35:24.746 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-02T04:35:24.746 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-04-02T04:35:24.747 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: [mgr] 2024-04-02T04:35:24.747 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-02T04:35:24.747 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 2024-04-02T04:35:24.747 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: [osd] 2024-04-02T04:35:24.747 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-02T04:35:24.747 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-02T04:35:25.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:24 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/2474648364' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-02T04:35:25.258 INFO:teuthology.orchestra.run.smithi053.stderr:Enabling cephadm module... 2024-04-02T04:35:26.236 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:25 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/888891075' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-02T04:35:27.146 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:26 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/888891075' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-02T04:35:27.146 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:26 smithi053 ceph-mon[24889]: mgrmap e5: smithi053.legdmv(active, since 15s) 2024-04-02T04:35:27.147 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: { 2024-04-02T04:35:27.147 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 5, 2024-04-02T04:35:27.147 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "available": true, 2024-04-02T04:35:27.147 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "active_name": "smithi053.legdmv", 2024-04-02T04:35:27.147 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-02T04:35:27.147 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: } 2024-04-02T04:35:27.652 INFO:teuthology.orchestra.run.smithi053.stderr:Waiting for the mgr to restart... 2024-04-02T04:35:27.652 INFO:teuthology.orchestra.run.smithi053.stderr:Waiting for mgr epoch 5... 2024-04-02T04:35:28.021 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:27 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/1906703004' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-02T04:35:32.967 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: { 2024-04-02T04:35:32.967 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-04-02T04:35:32.967 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "initialized": true 2024-04-02T04:35:32.967 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: } 2024-04-02T04:35:33.263 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:32 smithi053 ceph-mon[24889]: Active manager daemon smithi053.legdmv restarted 2024-04-02T04:35:33.263 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:32 smithi053 ceph-mon[24889]: Activating manager daemon smithi053.legdmv 2024-04-02T04:35:33.263 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:32 smithi053 ceph-mon[24889]: osdmap e2: 0 total, 0 up, 0 in 2024-04-02T04:35:33.263 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:32 smithi053 ceph-mon[24889]: mgrmap e6: smithi053.legdmv(active, starting, since 0.0525362s) 2024-04-02T04:35:33.264 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:32 smithi053 ceph-mon[24889]: from='mgr.14120 172.21.15.53:0/2225002104' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi053"}]: dispatch 2024-04-02T04:35:33.264 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:32 smithi053 ceph-mon[24889]: from='mgr.14120 172.21.15.53:0/2225002104' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mgr metadata", "who": "smithi053.legdmv", "id": "smithi053.legdmv"}]: dispatch 2024-04-02T04:35:33.264 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:32 smithi053 ceph-mon[24889]: from='mgr.14120 172.21.15.53:0/2225002104' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-02T04:35:33.264 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:32 smithi053 ceph-mon[24889]: from='mgr.14120 172.21.15.53:0/2225002104' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-02T04:35:33.264 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:32 smithi053 ceph-mon[24889]: from='mgr.14120 172.21.15.53:0/2225002104' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-02T04:35:33.264 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:32 smithi053 ceph-mon[24889]: Manager daemon smithi053.legdmv is now available 2024-04-02T04:35:33.264 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:32 smithi053 ceph-mon[24889]: from='mgr.14120 172.21.15.53:0/2225002104' entity='mgr.smithi053.legdmv' 2024-04-02T04:35:33.264 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:32 smithi053 ceph-mon[24889]: from='mgr.14120 172.21.15.53:0/2225002104' entity='mgr.smithi053.legdmv' 2024-04-02T04:35:33.264 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:32 smithi053 ceph-mon[24889]: from='mgr.14120 172.21.15.53:0/2225002104' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:35:33.264 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:32 smithi053 ceph-mon[24889]: from='mgr.14120 172.21.15.53:0/2225002104' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:35:33.264 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:32 smithi053 ceph-mon[24889]: from='mgr.14120 172.21.15.53:0/2225002104' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:35:33.264 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:32 smithi053 ceph-mon[24889]: from='mgr.14120 172.21.15.53:0/2225002104' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:35:33.264 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:32 smithi053 ceph-mon[24889]: from='mgr.14120 172.21.15.53:0/2225002104' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/trash_purge_schedule"}]: dispatch 2024-04-02T04:35:33.558 INFO:teuthology.orchestra.run.smithi053.stderr:mgr epoch 5 is available 2024-04-02T04:35:33.558 INFO:teuthology.orchestra.run.smithi053.stderr:Setting orchestrator backend to cephadm... 2024-04-02T04:35:34.144 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:33 smithi053 ceph-mon[24889]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-02T04:35:34.145 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:33 smithi053 ceph-mon[24889]: mgrmap e7: smithi053.legdmv(active, since 1.05462s) 2024-04-02T04:35:34.145 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:33 smithi053 ceph-mon[24889]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-02T04:35:35.039 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:34 smithi053 ceph-mon[24889]: mgrmap e8: smithi053.legdmv(active, since 2s) 2024-04-02T04:35:35.039 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:34 smithi053 ceph-mon[24889]: from='mgr.14120 172.21.15.53:0/2225002104' entity='mgr.smithi053.legdmv' 2024-04-02T04:35:35.039 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:34 smithi053 ceph-mon[24889]: from='mgr.14120 172.21.15.53:0/2225002104' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:35:35.040 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:34 smithi053 ceph-mon[24889]: from='mgr.14120 172.21.15.53:0/2225002104' entity='mgr.smithi053.legdmv' 2024-04-02T04:35:35.040 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:34 smithi053 ceph-mon[24889]: from='mgr.14120 172.21.15.53:0/2225002104' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:35:35.225 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: value unchanged 2024-04-02T04:35:35.686 INFO:teuthology.orchestra.run.smithi053.stderr:Generating ssh key... 2024-04-02T04:35:36.053 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:35 smithi053 ceph-mon[24889]: [02/Apr/2024:04:35:34] ENGINE Bus STARTING 2024-04-02T04:35:36.053 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:35 smithi053 ceph-mon[24889]: [02/Apr/2024:04:35:34] ENGINE Serving on https://172.21.15.53:7150 2024-04-02T04:35:36.053 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:35 smithi053 ceph-mon[24889]: [02/Apr/2024:04:35:34] ENGINE Bus STARTED 2024-04-02T04:35:37.281 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:36 smithi053 ceph-mon[24889]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:35:37.281 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:36 smithi053 ceph-mon[24889]: from='mgr.14120 172.21.15.53:0/2225002104' entity='mgr.smithi053.legdmv' 2024-04-02T04:35:37.281 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:36 smithi053 ceph-mon[24889]: from='mgr.14120 172.21.15.53:0/2225002104' entity='mgr.smithi053.legdmv' 2024-04-02T04:35:37.854 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkYsHriC9oJomtminNRKNXEhesrmpEgCmQ0BAGmU1jGLCtQl2Q26miXD6rgWMFhpjo+JbxmCCmx1c0VchtqnbdxltShUg4W/EQYqPS+ywqQNPBUEmHhsYHqr5M9yUOwTV8S1MtVZHxB6kwq4r1Fm0gLGAX2tpci4RXKUoT6KRYaQSWsJxPaerB9BNs/ML6XI5tzY4B29oa16jViEQGnfzRxUv6c6E9Zp5M7Q6HDeOLBrfTPfohRYSE4g4lvICcnjnxRHQPRAo2lCDQOOwB6CzaG85Tq7pYEPtVdYQR/P8vjr1YcdNgGXwBhga1wEEewiquX5HFn7ThtiUEvGhgKN2Z1RCNUnX8l1DVimUf5B4gr2zslPXqXuGVEAX9OllXEDjb7BD9SKnpwxVWBwc8pDf16hyXjoG9DrXY3NWk5BF2YGo37nb7g/CCSMVYDT3cuNtVignsHjSeLENoZif/lcg3Q7njfu5EZEqy3sHDPazFFY33qM2g+bELowRVDV4OWok= ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4 2024-04-02T04:35:38.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:37 smithi053 ceph-mon[24889]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:35:38.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:37 smithi053 ceph-mon[24889]: Generating ssh key... 2024-04-02T04:35:38.424 INFO:teuthology.orchestra.run.smithi053.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-02T04:35:38.424 INFO:teuthology.orchestra.run.smithi053.stderr:Adding key to root@localhost authorized_keys... 2024-04-02T04:35:38.424 INFO:teuthology.orchestra.run.smithi053.stderr:Adding host smithi053... 2024-04-02T04:35:39.214 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:38 smithi053 ceph-mon[24889]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:35:39.988 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: Added host 'smithi053' with addr '172.21.15.53' 2024-04-02T04:35:40.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:40 smithi053 ceph-mon[24889]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi053", "addr": "172.21.15.53", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:35:40.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:40 smithi053 ceph-mon[24889]: Deploying cephadm binary to smithi053 2024-04-02T04:35:40.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:40 smithi053 ceph-mon[24889]: from='mgr.14120 172.21.15.53:0/2225002104' entity='mgr.smithi053.legdmv' 2024-04-02T04:35:40.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:40 smithi053 ceph-mon[24889]: from='mgr.14120 172.21.15.53:0/2225002104' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:35:40.554 INFO:teuthology.orchestra.run.smithi053.stderr:Deploying mon service with default placement... 2024-04-02T04:35:41.195 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: Scheduled mon update... 2024-04-02T04:35:41.206 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:41 smithi053 ceph-mon[24889]: Added host smithi053 2024-04-02T04:35:42.098 INFO:teuthology.orchestra.run.smithi053.stderr:Deploying mgr service with default placement... 2024-04-02T04:35:42.365 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:42 smithi053 ceph-mon[24889]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:35:42.365 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:42 smithi053 ceph-mon[24889]: Saving service mon spec with placement count:5 2024-04-02T04:35:42.365 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:42 smithi053 ceph-mon[24889]: from='mgr.14120 172.21.15.53:0/2225002104' entity='mgr.smithi053.legdmv' 2024-04-02T04:35:42.670 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-04-02T04:35:43.699 INFO:teuthology.orchestra.run.smithi053.stderr:Deploying crash service with default placement... 2024-04-02T04:35:43.805 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:43 smithi053 ceph-mon[24889]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:35:43.806 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:43 smithi053 ceph-mon[24889]: Saving service mgr spec with placement count:2 2024-04-02T04:35:43.806 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:43 smithi053 ceph-mon[24889]: from='mgr.14120 172.21.15.53:0/2225002104' entity='mgr.smithi053.legdmv' 2024-04-02T04:35:43.806 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:43 smithi053 ceph-mon[24889]: from='mgr.14120 172.21.15.53:0/2225002104' entity='mgr.smithi053.legdmv' 2024-04-02T04:35:44.312 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: Scheduled crash update... 2024-04-02T04:35:44.897 INFO:teuthology.orchestra.run.smithi053.stderr:Deploying prometheus service with default placement... 2024-04-02T04:35:45.501 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:45 smithi053 ceph-mon[24889]: from='mgr.14120 172.21.15.53:0/2225002104' entity='mgr.smithi053.legdmv' 2024-04-02T04:35:45.502 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:45 smithi053 ceph-mon[24889]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:35:45.502 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:45 smithi053 ceph-mon[24889]: Saving service crash spec with placement * 2024-04-02T04:35:45.502 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:45 smithi053 ceph-mon[24889]: from='mgr.14120 172.21.15.53:0/2225002104' entity='mgr.smithi053.legdmv' 2024-04-02T04:35:45.502 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-04-02T04:35:46.695 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:46 smithi053 ceph-mon[24889]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:35:46.695 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:46 smithi053 ceph-mon[24889]: Saving service prometheus spec with placement count:1 2024-04-02T04:35:46.695 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:46 smithi053 ceph-mon[24889]: from='mgr.14120 172.21.15.53:0/2225002104' entity='mgr.smithi053.legdmv' 2024-04-02T04:35:46.695 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:46 smithi053 ceph-mon[24889]: from='mgr.14120 172.21.15.53:0/2225002104' entity='mgr.smithi053.legdmv' 2024-04-02T04:35:46.697 INFO:teuthology.orchestra.run.smithi053.stderr:Deploying grafana service with default placement... 2024-04-02T04:35:47.360 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-04-02T04:35:47.861 INFO:teuthology.orchestra.run.smithi053.stderr:Deploying node-exporter service with default placement... 2024-04-02T04:35:48.263 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:48 smithi053 ceph-mon[24889]: from='mgr.14120 172.21.15.53:0/2225002104' entity='mgr.smithi053.legdmv' 2024-04-02T04:35:48.263 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:48 smithi053 ceph-mon[24889]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:35:48.263 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:48 smithi053 ceph-mon[24889]: Saving service grafana spec with placement count:1 2024-04-02T04:35:48.263 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:48 smithi053 ceph-mon[24889]: from='mgr.14120 172.21.15.53:0/2225002104' entity='mgr.smithi053.legdmv' 2024-04-02T04:35:48.485 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-04-02T04:35:49.053 INFO:teuthology.orchestra.run.smithi053.stderr:Deploying alertmanager service with default placement... 2024-04-02T04:35:49.604 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:49 smithi053 ceph-mon[24889]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:35:49.604 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:49 smithi053 ceph-mon[24889]: Saving service node-exporter spec with placement * 2024-04-02T04:35:49.604 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:49 smithi053 ceph-mon[24889]: from='mgr.14120 172.21.15.53:0/2225002104' entity='mgr.smithi053.legdmv' 2024-04-02T04:35:49.650 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-04-02T04:35:50.786 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:50 smithi053 ceph-mon[24889]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:35:50.786 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:50 smithi053 ceph-mon[24889]: Saving service alertmanager spec with placement count:1 2024-04-02T04:35:50.786 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:50 smithi053 ceph-mon[24889]: from='mgr.14120 172.21.15.53:0/2225002104' entity='mgr.smithi053.legdmv' 2024-04-02T04:35:52.034 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:52 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/2092291000' entity='client.admin' 2024-04-02T04:35:52.035 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:52 smithi053 ceph-mon[24889]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:52.959 INFO:teuthology.orchestra.run.smithi053.stderr:Enabling the dashboard module... 2024-04-02T04:35:53.556 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:53 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/837979857' entity='client.admin' 2024-04-02T04:35:54.746 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:54 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/380681656' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-02T04:35:54.746 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:54 smithi053 ceph-mon[24889]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:35:55.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:55 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/380681656' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-02T04:35:55.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:55 smithi053 ceph-mon[24889]: mgrmap e9: smithi053.legdmv(active, since 22s) 2024-04-02T04:35:55.684 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: { 2024-04-02T04:35:55.684 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "epoch": 9, 2024-04-02T04:35:55.684 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "available": true, 2024-04-02T04:35:55.684 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "active_name": "smithi053.legdmv", 2024-04-02T04:35:55.684 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-02T04:35:55.684 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: } 2024-04-02T04:35:56.346 INFO:teuthology.orchestra.run.smithi053.stderr:Waiting for the mgr to restart... 2024-04-02T04:35:56.346 INFO:teuthology.orchestra.run.smithi053.stderr:Waiting for mgr epoch 9... 2024-04-02T04:35:56.524 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:35:56 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/486294879' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-02T04:36:01.483 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: { 2024-04-02T04:36:01.483 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-04-02T04:36:01.483 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: "initialized": true 2024-04-02T04:36:01.483 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: } 2024-04-02T04:36:01.777 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:01 smithi053 ceph-mon[24889]: Active manager daemon smithi053.legdmv restarted 2024-04-02T04:36:01.777 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:01 smithi053 ceph-mon[24889]: Activating manager daemon smithi053.legdmv 2024-04-02T04:36:01.778 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:01 smithi053 ceph-mon[24889]: osdmap e3: 0 total, 0 up, 0 in 2024-04-02T04:36:01.778 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:01 smithi053 ceph-mon[24889]: mgrmap e10: smithi053.legdmv(active, starting, since 0.0527257s) 2024-04-02T04:36:01.778 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:01 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi053"}]: dispatch 2024-04-02T04:36:01.778 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:01 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mgr metadata", "who": "smithi053.legdmv", "id": "smithi053.legdmv"}]: dispatch 2024-04-02T04:36:01.778 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:01 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-02T04:36:01.778 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:01 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-02T04:36:01.778 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:01 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-02T04:36:01.778 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:01 smithi053 ceph-mon[24889]: Manager daemon smithi053.legdmv is now available 2024-04-02T04:36:01.778 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:01 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:36:01.778 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:01 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:36:01.778 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:01 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:36:01.778 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:01 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/trash_purge_schedule"}]: dispatch 2024-04-02T04:36:02.031 INFO:teuthology.orchestra.run.smithi053.stderr:mgr epoch 9 is available 2024-04-02T04:36:02.031 INFO:teuthology.orchestra.run.smithi053.stderr:Generating a dashboard self-signed certificate... 2024-04-02T04:36:02.592 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:02 smithi053 ceph-mon[24889]: mgrmap e11: smithi053.legdmv(active, since 1.05481s) 2024-04-02T04:36:02.593 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:02 smithi053 ceph-mon[24889]: [02/Apr/2024:04:36:02] ENGINE Bus STARTING 2024-04-02T04:36:02.712 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: Self-signed certificate created 2024-04-02T04:36:03.337 INFO:teuthology.orchestra.run.smithi053.stderr:Creating initial admin user... 2024-04-02T04:36:03.746 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:03 smithi053 ceph-mon[24889]: [02/Apr/2024:04:36:02] ENGINE Serving on https://172.21.15.53:7150 2024-04-02T04:36:03.746 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:03 smithi053 ceph-mon[24889]: [02/Apr/2024:04:36:02] ENGINE Bus STARTED 2024-04-02T04:36:03.746 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:03 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' 2024-04-02T04:36:03.746 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:03 smithi053 ceph-mon[24889]: mgrmap e12: smithi053.legdmv(active, since 2s) 2024-04-02T04:36:03.746 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:03 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' 2024-04-02T04:36:03.746 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:03 smithi053 ceph-mon[24889]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:36:03.746 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:03 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' 2024-04-02T04:36:03.747 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:03 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' 2024-04-02T04:36:04.211 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$UKJtoXaYuYn71fWMlDTPFOLMo4HieRuSeOfWJqRFy9VBicuBxIQ7W", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1712032564, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-02T04:36:04.802 INFO:teuthology.orchestra.run.smithi053.stderr:Fetching dashboard port number... 2024-04-02T04:36:05.387 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:05 smithi053 ceph-mon[24889]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:36:05.387 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:05 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' 2024-04-02T04:36:05.387 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:05 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' 2024-04-02T04:36:05.437 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: 8443 2024-04-02T04:36:06.069 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: disabled 2024-04-02T04:36:06.076 INFO:teuthology.orchestra.run.smithi053.stderr:systemctl: inactive 2024-04-02T04:36:06.076 INFO:teuthology.orchestra.run.smithi053.stderr:firewalld.service is not enabled 2024-04-02T04:36:06.076 INFO:teuthology.orchestra.run.smithi053.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-02T04:36:06.078 INFO:teuthology.orchestra.run.smithi053.stderr:Ceph Dashboard is now available at: 2024-04-02T04:36:06.079 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-04-02T04:36:06.079 INFO:teuthology.orchestra.run.smithi053.stderr: URL: https://smithi053.front.sepia.ceph.com:8443/ 2024-04-02T04:36:06.079 INFO:teuthology.orchestra.run.smithi053.stderr: User: admin 2024-04-02T04:36:06.079 INFO:teuthology.orchestra.run.smithi053.stderr: Password: 6rnjwo5f3x 2024-04-02T04:36:06.079 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-04-02T04:36:06.079 INFO:teuthology.orchestra.run.smithi053.stderr:Enabling autotune for osd_memory_target 2024-04-02T04:36:06.331 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:06 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/3689686458' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-02T04:36:08.319 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:08 smithi053 ceph-mon[24889]: mgrmap e13: smithi053.legdmv(active, since 6s) 2024-04-02T04:36:08.840 INFO:teuthology.orchestra.run.smithi053.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-04-02T04:36:09.462 INFO:teuthology.orchestra.run.smithi053.stderr:You can access the Ceph CLI with: 2024-04-02T04:36:09.462 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-04-02T04:36:09.462 INFO:teuthology.orchestra.run.smithi053.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-02T04:36:09.462 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-04-02T04:36:09.462 INFO:teuthology.orchestra.run.smithi053.stderr:Please consider enabling telemetry to help improve Ceph: 2024-04-02T04:36:09.462 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-04-02T04:36:09.463 INFO:teuthology.orchestra.run.smithi053.stderr: ceph telemetry on 2024-04-02T04:36:09.463 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-04-02T04:36:09.463 INFO:teuthology.orchestra.run.smithi053.stderr:For more information see: 2024-04-02T04:36:09.463 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-04-02T04:36:09.463 INFO:teuthology.orchestra.run.smithi053.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-04-02T04:36:09.463 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-04-02T04:36:09.463 INFO:teuthology.orchestra.run.smithi053.stderr:Bootstrap complete. 2024-04-02T04:36:09.503 INFO:tasks.cephadm:Fetching config... 2024-04-02T04:36:09.504 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-02T04:36:09.504 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-02T04:36:09.522 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-02T04:36:09.522 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-02T04:36:09.522 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-02T04:36:09.577 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-02T04:36:09.577 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-02T04:36:09.577 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/mon.smithi053/keyring of=/dev/stdout 2024-04-02T04:36:09.645 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-02T04:36:09.645 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-02T04:36:09.645 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-02T04:36:09.700 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-02T04:36:09.701 DEBUG:teuthology.orchestra.run.smithi053:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkYsHriC9oJomtminNRKNXEhesrmpEgCmQ0BAGmU1jGLCtQl2Q26miXD6rgWMFhpjo+JbxmCCmx1c0VchtqnbdxltShUg4W/EQYqPS+ywqQNPBUEmHhsYHqr5M9yUOwTV8S1MtVZHxB6kwq4r1Fm0gLGAX2tpci4RXKUoT6KRYaQSWsJxPaerB9BNs/ML6XI5tzY4B29oa16jViEQGnfzRxUv6c6E9Zp5M7Q6HDeOLBrfTPfohRYSE4g4lvICcnjnxRHQPRAo2lCDQOOwB6CzaG85Tq7pYEPtVdYQR/P8vjr1YcdNgGXwBhga1wEEewiquX5HFn7ThtiUEvGhgKN2Z1RCNUnX8l1DVimUf5B4gr2zslPXqXuGVEAX9OllXEDjb7BD9SKnpwxVWBwc8pDf16hyXjoG9DrXY3NWk5BF2YGo37nb7g/CCSMVYDT3cuNtVignsHjSeLENoZif/lcg3Q7njfu5EZEqy3sHDPazFFY33qM2g+bELowRVDV4OWok= ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-02T04:36:09.781 INFO:teuthology.orchestra.run.smithi053.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkYsHriC9oJomtminNRKNXEhesrmpEgCmQ0BAGmU1jGLCtQl2Q26miXD6rgWMFhpjo+JbxmCCmx1c0VchtqnbdxltShUg4W/EQYqPS+ywqQNPBUEmHhsYHqr5M9yUOwTV8S1MtVZHxB6kwq4r1Fm0gLGAX2tpci4RXKUoT6KRYaQSWsJxPaerB9BNs/ML6XI5tzY4B29oa16jViEQGnfzRxUv6c6E9Zp5M7Q6HDeOLBrfTPfohRYSE4g4lvICcnjnxRHQPRAo2lCDQOOwB6CzaG85Tq7pYEPtVdYQR/P8vjr1YcdNgGXwBhga1wEEewiquX5HFn7ThtiUEvGhgKN2Z1RCNUnX8l1DVimUf5B4gr2zslPXqXuGVEAX9OllXEDjb7BD9SKnpwxVWBwc8pDf16hyXjoG9DrXY3NWk5BF2YGo37nb7g/CCSMVYDT3cuNtVignsHjSeLENoZif/lcg3Q7njfu5EZEqy3sHDPazFFY33qM2g+bELowRVDV4OWok= ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4 2024-04-02T04:36:09.799 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkYsHriC9oJomtminNRKNXEhesrmpEgCmQ0BAGmU1jGLCtQl2Q26miXD6rgWMFhpjo+JbxmCCmx1c0VchtqnbdxltShUg4W/EQYqPS+ywqQNPBUEmHhsYHqr5M9yUOwTV8S1MtVZHxB6kwq4r1Fm0gLGAX2tpci4RXKUoT6KRYaQSWsJxPaerB9BNs/ML6XI5tzY4B29oa16jViEQGnfzRxUv6c6E9Zp5M7Q6HDeOLBrfTPfohRYSE4g4lvICcnjnxRHQPRAo2lCDQOOwB6CzaG85Tq7pYEPtVdYQR/P8vjr1YcdNgGXwBhga1wEEewiquX5HFn7ThtiUEvGhgKN2Z1RCNUnX8l1DVimUf5B4gr2zslPXqXuGVEAX9OllXEDjb7BD9SKnpwxVWBwc8pDf16hyXjoG9DrXY3NWk5BF2YGo37nb7g/CCSMVYDT3cuNtVignsHjSeLENoZif/lcg3Q7njfu5EZEqy3sHDPazFFY33qM2g+bELowRVDV4OWok= ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-02T04:36:09.844 INFO:teuthology.orchestra.run.smithi122.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkYsHriC9oJomtminNRKNXEhesrmpEgCmQ0BAGmU1jGLCtQl2Q26miXD6rgWMFhpjo+JbxmCCmx1c0VchtqnbdxltShUg4W/EQYqPS+ywqQNPBUEmHhsYHqr5M9yUOwTV8S1MtVZHxB6kwq4r1Fm0gLGAX2tpci4RXKUoT6KRYaQSWsJxPaerB9BNs/ML6XI5tzY4B29oa16jViEQGnfzRxUv6c6E9Zp5M7Q6HDeOLBrfTPfohRYSE4g4lvICcnjnxRHQPRAo2lCDQOOwB6CzaG85Tq7pYEPtVdYQR/P8vjr1YcdNgGXwBhga1wEEewiquX5HFn7ThtiUEvGhgKN2Z1RCNUnX8l1DVimUf5B4gr2zslPXqXuGVEAX9OllXEDjb7BD9SKnpwxVWBwc8pDf16hyXjoG9DrXY3NWk5BF2YGo37nb7g/CCSMVYDT3cuNtVignsHjSeLENoZif/lcg3Q7njfu5EZEqy3sHDPazFFY33qM2g+bELowRVDV4OWok= ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4 2024-04-02T04:36:09.859 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-02T04:36:10.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:09 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/226979885' entity='client.admin' 2024-04-02T04:36:12.653 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:12 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' 2024-04-02T04:36:12.654 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:12 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd/host:smithi053", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:36:12.654 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:12 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' 2024-04-02T04:36:13.481 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-02T04:36:13.481 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-02T04:36:13.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:13 smithi053 ceph-mon[24889]: Deploying daemon alertmanager.smithi053 on smithi053 2024-04-02T04:36:13.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:13 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/3754887964' entity='client.admin' 2024-04-02T04:36:16.881 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi122 2024-04-02T04:36:16.882 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-02T04:36:16.882 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/etc/ceph/ceph.conf 2024-04-02T04:36:16.900 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-02T04:36:16.900 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-02T04:36:16.957 INFO:tasks.cephadm:Adding host smithi122 to orchestrator... 2024-04-02T04:36:16.958 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph orch host add smithi122 2024-04-02T04:36:17.273 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:16 smithi053 ceph-mon[24889]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:36:17.273 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:16 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' 2024-04-02T04:36:21.544 INFO:teuthology.orchestra.run.smithi053.stdout:Added host 'smithi122' with addr '172.21.15.122' 2024-04-02T04:36:21.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:21 smithi053 ceph-mon[24889]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:21.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:21 smithi053 ceph-mon[24889]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi122", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:36:21.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:21 smithi053 ceph-mon[24889]: Deploying cephadm binary to smithi122 2024-04-02T04:36:22.115 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph orch host ls --format=json 2024-04-02T04:36:22.734 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:22 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' 2024-04-02T04:36:22.734 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:22 smithi053 ceph-mon[24889]: Added host smithi122 2024-04-02T04:36:24.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:23 smithi053 ceph-mon[24889]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:24.273 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-02T04:36:24.274 INFO:teuthology.orchestra.run.smithi053.stdout:[{"addr": "172.21.15.53", "hostname": "smithi053", "labels": [], "status": ""}, {"addr": "172.21.15.122", "hostname": "smithi122", "labels": [], "status": ""}] 2024-04-02T04:36:25.162 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:24 smithi053 ceph-mon[24889]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:36:25.481 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-02T04:36:25.481 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph osd crush tunables default 2024-04-02T04:36:26.390 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:26 smithi053 ceph-mon[24889]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:26.390 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:26 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' 2024-04-02T04:36:26.390 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:26 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi053", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-02T04:36:26.390 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:26 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi053", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-02T04:36:26.390 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:26 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:26.390 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:26 smithi053 ceph-mon[24889]: Deploying daemon crash.smithi053 on smithi053 2024-04-02T04:36:26.391 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:26 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' 2024-04-02T04:36:28.129 INFO:teuthology.orchestra.run.smithi053.stderr:adjusted tunables profile to default 2024-04-02T04:36:28.427 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:28 smithi053 ceph-mon[24889]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:28.428 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:28 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/899534201' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-02T04:36:28.708 INFO:tasks.cephadm:Adding mon.smithi053 on smithi053 2024-04-02T04:36:28.708 INFO:tasks.cephadm:Adding mon.smithi122 on smithi122 2024-04-02T04:36:28.708 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph orch apply mon '2;smithi053:172.21.15.53=smithi053;smithi122:172.21.15.122=smithi122' 2024-04-02T04:36:29.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:29 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/899534201' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-02T04:36:29.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:29 smithi053 ceph-mon[24889]: osdmap e4: 0 total, 0 up, 0 in 2024-04-02T04:36:30.252 INFO:teuthology.orchestra.run.smithi122.stdout:Scheduled mon update... 2024-04-02T04:36:30.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:30 smithi053 ceph-mon[24889]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:30.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:30 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' 2024-04-02T04:36:30.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:30 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' 2024-04-02T04:36:30.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:30 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' 2024-04-02T04:36:30.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:30 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-02T04:36:30.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:30 smithi053 ceph-mon[24889]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-02T04:36:30.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:30 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' 2024-04-02T04:36:30.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:30 smithi053 ceph-mon[24889]: Deploying daemon grafana.smithi053 on smithi053 2024-04-02T04:36:30.778 DEBUG:teuthology.orchestra.run.smithi122:mon.smithi122> sudo journalctl -f -n 0 -u ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@mon.smithi122.service 2024-04-02T04:36:30.780 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-02T04:36:30.780 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-04-02T04:36:31.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:31 smithi053 ceph-mon[24889]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi053:172.21.15.53=smithi053;smithi122:172.21.15.122=smithi122", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:36:31.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:31 smithi053 ceph-mon[24889]: Saving service mon spec with placement smithi053:172.21.15.53=smithi053;smithi122:172.21.15.122=smithi122;count:2 2024-04-02T04:36:31.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:31 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' 2024-04-02T04:36:31.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:31 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' 2024-04-02T04:36:32.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:32 smithi053 ceph-mon[24889]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:32.582 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-02T04:36:32.582 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"48e16742-f0aa-11ee-b647-cb9ed24678a4","modified":"2024-04-02T04:34:55.992242Z","created":"2024-04-02T04:34:55.992242Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-02T04:36:32.582 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-04-02T04:36:33.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:33 smithi053 ceph-mon[24889]: from='client.? 172.21.15.122:0/3612959105' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-02T04:36:34.022 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-02T04:36:34.023 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-04-02T04:36:34.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:34 smithi053 ceph-mon[24889]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:35.665 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-02T04:36:35.665 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"48e16742-f0aa-11ee-b647-cb9ed24678a4","modified":"2024-04-02T04:34:55.992242Z","created":"2024-04-02T04:34:55.992242Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-02T04:36:35.666 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-04-02T04:36:36.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:36 smithi053 ceph-mon[24889]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:36.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:36 smithi053 ceph-mon[24889]: from='client.? 172.21.15.122:0/3176773785' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-02T04:36:37.252 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-02T04:36:37.252 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-04-02T04:36:38.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:38 smithi053 ceph-mon[24889]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:38.959 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-02T04:36:38.960 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"48e16742-f0aa-11ee-b647-cb9ed24678a4","modified":"2024-04-02T04:34:55.992242Z","created":"2024-04-02T04:34:55.992242Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-02T04:36:38.961 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-04-02T04:36:39.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:39 smithi053 ceph-mon[24889]: from='client.? 172.21.15.122:0/40281545' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-02T04:36:40.403 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-02T04:36:40.404 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-04-02T04:36:40.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:40 smithi053 ceph-mon[24889]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:42.318 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-02T04:36:42.318 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"48e16742-f0aa-11ee-b647-cb9ed24678a4","modified":"2024-04-02T04:34:55.992242Z","created":"2024-04-02T04:34:55.992242Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-02T04:36:42.319 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-04-02T04:36:42.451 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:42 smithi053 ceph-mon[24889]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:42.452 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:42 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' 2024-04-02T04:36:43.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:43 smithi053 ceph-mon[24889]: Deploying daemon node-exporter.smithi053 on smithi053 2024-04-02T04:36:43.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:43 smithi053 ceph-mon[24889]: from='client.? 172.21.15.122:0/2397237631' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-02T04:36:43.899 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-02T04:36:43.900 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-04-02T04:36:44.369 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:44 smithi053 ceph-mon[24889]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:36:45.599 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-02T04:36:45.599 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"48e16742-f0aa-11ee-b647-cb9ed24678a4","modified":"2024-04-02T04:34:55.992242Z","created":"2024-04-02T04:34:55.992242Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-02T04:36:45.601 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-04-02T04:36:45.806 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:45 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' 2024-04-02T04:36:45.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:45 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-02T04:36:46.731 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:46 smithi053 ceph-mon[24889]: from='mgr.14162 172.21.15.53:0/122995597' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-02T04:36:46.731 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:46 smithi053 ceph-mon[24889]: mgrmap e14: smithi053.legdmv(active, since 45s) 2024-04-02T04:36:46.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:46 smithi053 ceph-mon[24889]: from='client.? 172.21.15.122:0/182429985' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-02T04:36:47.136 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-02T04:36:47.137 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-04-02T04:36:48.838 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-02T04:36:48.838 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"48e16742-f0aa-11ee-b647-cb9ed24678a4","modified":"2024-04-02T04:34:55.992242Z","created":"2024-04-02T04:34:55.992242Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-02T04:36:48.840 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-04-02T04:36:49.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:48 smithi053 ceph-mon[24889]: from='client.? 172.21.15.122:0/1296822195' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-02T04:36:50.396 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-02T04:36:50.396 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-04-02T04:36:52.216 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-02T04:36:52.217 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"48e16742-f0aa-11ee-b647-cb9ed24678a4","modified":"2024-04-02T04:34:55.992242Z","created":"2024-04-02T04:34:55.992242Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-02T04:36:52.218 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-04-02T04:36:52.731 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:52 smithi053 ceph-mon[24889]: Active manager daemon smithi053.legdmv restarted 2024-04-02T04:36:52.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:52 smithi053 ceph-mon[24889]: Activating manager daemon smithi053.legdmv 2024-04-02T04:36:52.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:52 smithi053 ceph-mon[24889]: osdmap e5: 0 total, 0 up, 0 in 2024-04-02T04:36:52.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:52 smithi053 ceph-mon[24889]: mgrmap e15: smithi053.legdmv(active, starting, since 0.0525938s) 2024-04-02T04:36:52.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:52 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi053"}]: dispatch 2024-04-02T04:36:52.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:52 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mgr metadata", "who": "smithi053.legdmv", "id": "smithi053.legdmv"}]: dispatch 2024-04-02T04:36:52.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:52 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-02T04:36:52.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:52 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-02T04:36:52.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:52 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-02T04:36:52.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:52 smithi053 ceph-mon[24889]: Manager daemon smithi053.legdmv is now available 2024-04-02T04:36:52.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:52 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:36:52.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:52 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:36:52.733 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:52 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:36:52.733 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:52 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:36:52.733 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:52 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:36:52.733 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:52 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:36:52.733 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:52 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/trash_purge_schedule"}]: dispatch 2024-04-02T04:36:52.733 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:52 smithi053 ceph-mon[24889]: from='client.? 172.21.15.122:0/3140047895' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-02T04:36:52.733 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:52 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:36:53.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:53 smithi053 ceph-mon[24889]: mgrmap e16: smithi053.legdmv(active, since 1.05451s) 2024-04-02T04:36:53.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:53 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:36:53.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:53 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:36:53.878 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-02T04:36:53.879 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-04-02T04:36:54.663 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:54 smithi053 ceph-mon[24889]: [02/Apr/2024:04:36:52] ENGINE Bus STARTING 2024-04-02T04:36:54.664 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:54 smithi053 ceph-mon[24889]: [02/Apr/2024:04:36:52] ENGINE Serving on https://172.21.15.53:7150 2024-04-02T04:36:54.664 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:54 smithi053 ceph-mon[24889]: [02/Apr/2024:04:36:52] ENGINE Bus STARTED 2024-04-02T04:36:54.664 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:54 smithi053 ceph-mon[24889]: mgrmap e17: smithi053.legdmv(active, since 2s) 2024-04-02T04:36:54.664 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:54 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:36:56.414 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-02T04:36:56.414 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"48e16742-f0aa-11ee-b647-cb9ed24678a4","modified":"2024-04-02T04:34:55.992242Z","created":"2024-04-02T04:34:55.992242Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-02T04:36:56.416 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-04-02T04:36:56.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:56 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:36:56.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:56 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:36:56.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:56 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd/host:smithi053", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:36:57.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:57 smithi053 ceph-mon[24889]: Updating smithi053:/etc/ceph/ceph.conf 2024-04-02T04:36:57.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:57 smithi053 ceph-mon[24889]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-04-02T04:36:57.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:57 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:36:57.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:57 smithi053 ceph-mon[24889]: from='client.? 172.21.15.122:0/292765986' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-02T04:36:58.070 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-02T04:36:58.070 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-04-02T04:37:00.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:59 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:00.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:59 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:00.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:59 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd/host:smithi122", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:37:00.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:59 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:00.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:59 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi122", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-02T04:37:00.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:59 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi122", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-02T04:37:00.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:59 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:37:00.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:36:59 smithi053 ceph-mon[24889]: Deploying daemon crash.smithi122 on smithi122 2024-04-02T04:37:00.298 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-02T04:37:00.299 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"48e16742-f0aa-11ee-b647-cb9ed24678a4","modified":"2024-04-02T04:34:55.992242Z","created":"2024-04-02T04:34:55.992242Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-02T04:37:00.301 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-04-02T04:37:01.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:00 smithi053 ceph-mon[24889]: from='client.? 172.21.15.122:0/3307637142' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-02T04:37:01.899 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-02T04:37:01.900 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-04-02T04:37:03.837 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-02T04:37:03.838 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"48e16742-f0aa-11ee-b647-cb9ed24678a4","modified":"2024-04-02T04:34:55.992242Z","created":"2024-04-02T04:34:55.992242Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-02T04:37:03.839 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-04-02T04:37:04.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:03 smithi053 ceph-mon[24889]: from='client.? 172.21.15.122:0/2897185372' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-02T04:37:05.623 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-02T04:37:05.623 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-04-02T04:37:05.806 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:05 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:05.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:05 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi122.cpwlyk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-02T04:37:05.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:05 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi122.cpwlyk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-02T04:37:05.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:05 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-02T04:37:05.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:05 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:37:05.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:05 smithi053 ceph-mon[24889]: Deploying daemon mgr.smithi122.cpwlyk on smithi122 2024-04-02T04:37:07.527 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-02T04:37:07.527 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"48e16742-f0aa-11ee-b647-cb9ed24678a4","modified":"2024-04-02T04:34:55.992242Z","created":"2024-04-02T04:34:55.992242Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-02T04:37:07.528 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-04-02T04:37:07.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:07 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:08.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:08 smithi053 ceph-mon[24889]: from='client.? 172.21.15.122:0/1774036069' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-02T04:37:08.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:08 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:08.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:08 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T04:37:08.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:08 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:37:08.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:08 smithi053 ceph-mon[24889]: Deploying daemon mon.smithi122 on smithi122 2024-04-02T04:37:09.257 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-02T04:37:09.258 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-04-02T04:37:11.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:11 smithi053 ceph-mon[24889]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:11.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:11 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:12.369 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-02T04:37:12.369 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"48e16742-f0aa-11ee-b647-cb9ed24678a4","modified":"2024-04-02T04:34:55.992242Z","created":"2024-04-02T04:34:55.992242Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-02T04:37:12.370 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-04-02T04:37:12.806 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:12 smithi053 ceph-mon[24889]: from='client.? 172.21.15.122:0/1922765953' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-02T04:37:14.240 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-02T04:37:14.240 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-04-02T04:37:18.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:17 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-02T04:37:18.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:17 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi053"}]: dispatch 2024-04-02T04:37:18.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:17 smithi053 ceph-mon[24889]: mon.smithi053 calling monitor election 2024-04-02T04:37:18.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:17 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-02T04:37:18.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:17 smithi053 ceph-mon[24889]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:18.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:17 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-02T04:37:18.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:17 smithi053 ceph-mon[24889]: from='mgr.? 172.21.15.122:0/1529849869' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.cpwlyk/crt"}]: dispatch 2024-04-02T04:37:18.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:17 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-02T04:37:18.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:17 smithi053 ceph-mon[24889]: mon.smithi122 calling monitor election 2024-04-02T04:37:18.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:17 smithi053 ceph-mon[24889]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:18.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:17 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-02T04:37:18.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:17 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-02T04:37:18.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:17 smithi053 ceph-mon[24889]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:18.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:17 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-02T04:37:18.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:17 smithi053 ceph-mon[24889]: mon.smithi053 is new leader, mons smithi053,smithi122 in quorum (ranks 0,1) 2024-04-02T04:37:18.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:17 smithi053 ceph-mon[24889]: monmap e2: 2 mons at {smithi053=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0],smithi122=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0]} 2024-04-02T04:37:18.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:17 smithi053 ceph-mon[24889]: fsmap 2024-04-02T04:37:18.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:17 smithi053 ceph-mon[24889]: osdmap e5: 0 total, 0 up, 0 in 2024-04-02T04:37:18.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:17 smithi053 ceph-mon[24889]: mgrmap e17: smithi053.legdmv(active, since 26s) 2024-04-02T04:37:18.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:17 smithi053 ceph-mon[24889]: Standby manager daemon smithi122.cpwlyk started 2024-04-02T04:37:18.059 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:17 smithi053 ceph-mon[24889]: from='mgr.? 172.21.15.122:0/1529849869' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-02T04:37:18.059 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:17 smithi053 ceph-mon[24889]: overall HEALTH_OK 2024-04-02T04:37:18.059 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:17 smithi053 ceph-mon[24889]: from='mgr.? 172.21.15.122:0/1529849869' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.cpwlyk/key"}]: dispatch 2024-04-02T04:37:18.059 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:17 smithi053 ceph-mon[24889]: from='mgr.? 172.21.15.122:0/1529849869' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-02T04:37:18.059 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:17 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:18.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:18 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-02T04:37:18.969 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-02T04:37:18.970 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":2,"fsid":"48e16742-f0aa-11ee-b647-cb9ed24678a4","modified":"2024-04-02T04:37:12.545751Z","created":"2024-04-02T04:34:55.992242Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi053","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi122","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-02T04:37:18.970 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 2 2024-04-02T04:37:19.056 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:18 smithi053 ceph-mon[24889]: Deploying daemon node-exporter.smithi122 on smithi122 2024-04-02T04:37:19.056 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:18 smithi053 ceph-mon[24889]: mgrmap e18: smithi053.legdmv(active, since 26s), standbys: smithi122.cpwlyk 2024-04-02T04:37:19.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:18 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mgr metadata", "who": "smithi122.cpwlyk", "id": "smithi122.cpwlyk"}]: dispatch 2024-04-02T04:37:19.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:18 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-02T04:37:19.870 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-02T04:37:19.870 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph config generate-minimal-conf 2024-04-02T04:37:19.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:19 smithi122 ceph-mon[27145]: from='client.? 172.21.15.122:0/3783926765' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-02T04:37:19.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:19 smithi122 ceph-mon[27145]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:19.899 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:19 smithi053 ceph-mon[24889]: from='client.? 172.21.15.122:0/3783926765' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-02T04:37:19.899 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:19 smithi053 ceph-mon[24889]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:21.635 INFO:teuthology.orchestra.run.smithi053.stdout:# minimal ceph.conf for 48e16742-f0aa-11ee-b647-cb9ed24678a4 2024-04-02T04:37:21.635 INFO:teuthology.orchestra.run.smithi053.stdout:[global] 2024-04-02T04:37:21.636 INFO:teuthology.orchestra.run.smithi053.stdout: fsid = 48e16742-f0aa-11ee-b647-cb9ed24678a4 2024-04-02T04:37:21.636 INFO:teuthology.orchestra.run.smithi053.stdout: mon_host = [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-04-02T04:37:21.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:21 smithi053 ceph-mon[24889]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:21.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:21 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:21.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:21 smithi122 ceph-mon[27145]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:21.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:21 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:22.215 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-02T04:37:22.216 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-02T04:37:22.216 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/ceph/ceph.conf 2024-04-02T04:37:22.285 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-02T04:37:22.285 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-02T04:37:22.353 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-02T04:37:22.353 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.conf 2024-04-02T04:37:22.385 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-02T04:37:22.385 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-02T04:37:22.453 INFO:tasks.cephadm:Deploying OSDs... 2024-04-02T04:37:22.454 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-02T04:37:22.454 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/scratch_devs of=/dev/stdout 2024-04-02T04:37:22.468 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-02T04:37:22.468 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_1 2024-04-02T04:37:22.522 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-02T04:37:22.522 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T04:37:22.522 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-02T04:37:22.522 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T04:37:22.523 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T04:37:22.523 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-04-02 04:36:42.385016043 +0000 2024-04-02T04:37:22.523 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-04-02 04:34:00.230043678 +0000 2024-04-02T04:37:22.523 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-04-02 04:34:00.230043678 +0000 2024-04-02T04:37:22.523 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-04-02 04:34:00.230043678 +0000 2024-04-02T04:37:22.523 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-02T04:37:22.588 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-04-02T04:37:22.588 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-04-02T04:37:22.588 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000102429 s, 5.0 MB/s 2024-04-02T04:37:22.589 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-02T04:37:22.644 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_2 2024-04-02T04:37:22.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:22 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/229776917' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:37:22.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:22 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:22.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:22 smithi122 ceph-mon[27145]: Deploying daemon prometheus.smithi053 on smithi053 2024-04-02T04:37:22.698 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-02T04:37:22.698 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T04:37:22.698 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-02T04:37:22.698 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T04:37:22.699 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T04:37:22.699 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-04-02 04:36:42.386016023 +0000 2024-04-02T04:37:22.699 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-04-02 04:34:00.545037338 +0000 2024-04-02T04:37:22.699 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-04-02 04:34:00.545037338 +0000 2024-04-02T04:37:22.699 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-04-02 04:34:00.545037338 +0000 2024-04-02T04:37:22.699 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-02T04:37:22.763 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-04-02T04:37:22.763 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-04-02T04:37:22.763 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000105109 s, 4.9 MB/s 2024-04-02T04:37:22.764 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-02T04:37:22.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:22 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/229776917' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:37:22.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:22 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:22.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:22 smithi053 ceph-mon[24889]: Deploying daemon prometheus.smithi053 on smithi053 2024-04-02T04:37:22.819 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_3 2024-04-02T04:37:22.874 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-02T04:37:22.874 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T04:37:22.874 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-04-02T04:37:22.874 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T04:37:22.874 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T04:37:22.874 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-04-02 04:36:42.386016023 +0000 2024-04-02T04:37:22.875 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-04-02 04:34:00.895030292 +0000 2024-04-02T04:37:22.875 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-04-02 04:34:00.895030292 +0000 2024-04-02T04:37:22.875 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-04-02 04:34:00.895030292 +0000 2024-04-02T04:37:22.875 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-02T04:37:22.939 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-04-02T04:37:22.939 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-04-02T04:37:22.939 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 8.8444e-05 s, 5.8 MB/s 2024-04-02T04:37:22.940 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-02T04:37:22.995 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_4 2024-04-02T04:37:23.050 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-02T04:37:23.050 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T04:37:23.050 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-02T04:37:23.050 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T04:37:23.050 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T04:37:23.050 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-04-02 04:36:42.387016004 +0000 2024-04-02T04:37:23.050 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-04-02 04:34:01.221023730 +0000 2024-04-02T04:37:23.050 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-04-02 04:34:01.221023730 +0000 2024-04-02T04:37:23.050 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-04-02 04:34:01.221023730 +0000 2024-04-02T04:37:23.051 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-02T04:37:23.115 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-04-02T04:37:23.115 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-04-02T04:37:23.115 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.00010112 s, 5.1 MB/s 2024-04-02T04:37:23.116 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-02T04:37:23.171 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-02T04:37:23.171 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/scratch_devs of=/dev/stdout 2024-04-02T04:37:23.190 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-02T04:37:23.190 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_1 2024-04-02T04:37:23.247 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-02T04:37:23.247 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T04:37:23.247 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-02T04:37:23.247 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T04:37:23.247 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T04:37:23.247 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-04-02 04:37:18.164601491 +0000 2024-04-02T04:37:23.247 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-04-02 04:33:49.365644095 +0000 2024-04-02T04:37:23.248 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-04-02 04:33:49.365644095 +0000 2024-04-02T04:37:23.248 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-04-02 04:33:49.365644095 +0000 2024-04-02T04:37:23.248 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-02T04:37:23.316 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-04-02T04:37:23.316 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-04-02T04:37:23.316 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000231855 s, 2.2 MB/s 2024-04-02T04:37:23.317 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-02T04:37:23.374 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_2 2024-04-02T04:37:23.431 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-02T04:37:23.431 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T04:37:23.431 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-04-02T04:37:23.431 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T04:37:23.431 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T04:37:23.431 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-04-02 04:37:18.165601468 +0000 2024-04-02T04:37:23.431 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-04-02 04:33:49.724635022 +0000 2024-04-02T04:37:23.432 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-04-02 04:33:49.724635022 +0000 2024-04-02T04:37:23.432 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-04-02 04:33:49.724635022 +0000 2024-04-02T04:37:23.432 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-02T04:37:23.498 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-04-02T04:37:23.498 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-04-02T04:37:23.498 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000211286 s, 2.4 MB/s 2024-04-02T04:37:23.499 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-02T04:37:23.556 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_3 2024-04-02T04:37:23.611 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-02T04:37:23.611 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T04:37:23.611 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-02T04:37:23.611 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T04:37:23.612 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T04:37:23.612 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-04-02 04:37:18.165601468 +0000 2024-04-02T04:37:23.612 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-04-02 04:33:50.088625822 +0000 2024-04-02T04:37:23.612 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-04-02 04:33:50.088625822 +0000 2024-04-02T04:37:23.612 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-04-02 04:33:50.088625822 +0000 2024-04-02T04:37:23.612 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-02T04:37:23.678 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-04-02T04:37:23.678 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-04-02T04:37:23.678 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000263394 s, 1.9 MB/s 2024-04-02T04:37:23.679 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-02T04:37:23.735 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_4 2024-04-02T04:37:23.790 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-02T04:37:23.790 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T04:37:23.790 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-02T04:37:23.790 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T04:37:23.790 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T04:37:23.790 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-04-02 04:37:18.165601468 +0000 2024-04-02T04:37:23.791 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-04-02 04:33:50.429617204 +0000 2024-04-02T04:37:23.791 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-04-02 04:33:50.429617204 +0000 2024-04-02T04:37:23.791 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-04-02 04:33:50.429617204 +0000 2024-04-02T04:37:23.791 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-02T04:37:23.856 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-04-02T04:37:23.856 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-04-02T04:37:23.856 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000236822 s, 2.2 MB/s 2024-04-02T04:37:23.857 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-02T04:37:23.913 INFO:tasks.cephadm:Deploying osd.0 on smithi053 with /dev/vg_nvme/lv_4... 2024-04-02T04:37:23.913 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-02T04:37:24.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:23 smithi122 ceph-mon[27145]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:24.218 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:23 smithi053 ceph-mon[24889]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:25.792 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:25 smithi053 ceph-mon[24889]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:25.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:25 smithi122 ceph-mon[27145]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:27.245 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-02T04:37:27.245 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-02T04:37:27.245 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10+0 records in 2024-04-02T04:37:27.245 INFO:teuthology.orchestra.run.smithi053.stderr:10+0 records out 2024-04-02T04:37:27.245 INFO:teuthology.orchestra.run.smithi053.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0123227 s, 851 MB/s 2024-04-02T04:37:27.245 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping successful for: 2024-04-02T04:37:27.806 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:27 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:27.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:27 smithi053 ceph-mon[24889]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:27.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:27 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:27.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:27 smithi122 ceph-mon[27145]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:27.945 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi053:vg_nvme/lv_4 2024-04-02T04:37:29.742 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:29 smithi053 ceph-mon[24889]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:29.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:29 smithi122 ceph-mon[27145]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:31.728 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:31 smithi053 ceph-mon[24889]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:31.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:31 smithi122 ceph-mon[27145]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:34.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:33 smithi053 ceph-mon[24889]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:34.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:33 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:37:34.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:33 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:37:34.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:33 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:37:34.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:33 smithi122 ceph-mon[27145]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:34.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:33 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:37:34.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:33 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:37:34.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:33 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:37:35.287 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:34 smithi053 ceph-mon[24889]: from='client.14240 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:37:35.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:34 smithi122 ceph-mon[27145]: from='client.14240 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:37:36.490 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:36 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:36.490 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:36 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:37:36.491 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:36 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:37:36.491 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:36 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:37:36.491 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:36 smithi053 ceph-mon[24889]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:36.614 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:36 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:36.614 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:36 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:37:36.614 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:36 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:37:36.614 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:36 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:37:36.614 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:36 smithi122 ceph-mon[27145]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:37.796 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:37 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:37.796 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:37 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:37.796 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:37 smithi053 ceph-mon[24889]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-02T04:37:37.797 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:37 smithi053 ceph-mon[24889]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-02T04:37:37.797 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:37 smithi053 ceph-mon[24889]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:37.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:37 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:37.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:37 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:37.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:37 smithi122 ceph-mon[27145]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-02T04:37:37.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:37 smithi122 ceph-mon[27145]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-02T04:37:37.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:37 smithi122 ceph-mon[27145]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:38.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:38 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:38.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:38 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/1033272190' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98971ffe-6ad2-4adc-8d2d-7b19f4efc239"}]: dispatch 2024-04-02T04:37:38.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:38 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/1033272190' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98971ffe-6ad2-4adc-8d2d-7b19f4efc239"}]': finished 2024-04-02T04:37:38.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:38 smithi053 ceph-mon[24889]: osdmap e6: 1 total, 0 up, 1 in 2024-04-02T04:37:38.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:38 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:37:38.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:38 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/2129667262' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:37:38.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:38 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:38.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:38 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/1033272190' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98971ffe-6ad2-4adc-8d2d-7b19f4efc239"}]: dispatch 2024-04-02T04:37:38.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:38 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/1033272190' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98971ffe-6ad2-4adc-8d2d-7b19f4efc239"}]': finished 2024-04-02T04:37:38.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:38 smithi122 ceph-mon[27145]: osdmap e6: 1 total, 0 up, 1 in 2024-04-02T04:37:38.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:38 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:37:38.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:38 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/2129667262' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:37:39.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:39 smithi053 ceph-mon[24889]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:39.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:39 smithi122 ceph-mon[27145]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:41.427 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:41 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:41.427 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:41 smithi053 ceph-mon[24889]: Updating smithi053:/etc/ceph/ceph.conf 2024-04-02T04:37:41.427 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:41 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:41.427 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:41 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:41.427 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:41 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T04:37:41.427 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:41 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-02T04:37:41.428 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:41 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:37:41.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:41 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:41.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:41 smithi122 ceph-mon[27145]: Updating smithi053:/etc/ceph/ceph.conf 2024-04-02T04:37:41.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:41 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:41.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:41 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:41.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:41 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T04:37:41.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:41 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-02T04:37:41.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:41 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:37:42.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:42 smithi053 ceph-mon[24889]: Reconfiguring mon.smithi053 (unknown last config time)... 2024-04-02T04:37:42.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:42 smithi053 ceph-mon[24889]: Reconfiguring daemon mon.smithi053 on smithi053 2024-04-02T04:37:42.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:42 smithi053 ceph-mon[24889]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:42.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:42 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:42.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:42 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi053.legdmv", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-02T04:37:42.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:42 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-02T04:37:42.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:42 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:37:42.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:42 smithi122 ceph-mon[27145]: Reconfiguring mon.smithi053 (unknown last config time)... 2024-04-02T04:37:42.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:42 smithi122 ceph-mon[27145]: Reconfiguring daemon mon.smithi053 on smithi053 2024-04-02T04:37:42.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:42 smithi122 ceph-mon[27145]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:42.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:42 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:42.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:42 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi053.legdmv", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-02T04:37:42.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:42 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-02T04:37:42.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:42 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:37:43.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:43 smithi053 ceph-mon[24889]: Reconfiguring mgr.smithi053.legdmv (unknown last config time)... 2024-04-02T04:37:43.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:43 smithi053 ceph-mon[24889]: Reconfiguring daemon mgr.smithi053.legdmv on smithi053 2024-04-02T04:37:43.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:43 smithi122 ceph-mon[27145]: Reconfiguring mgr.smithi053.legdmv (unknown last config time)... 2024-04-02T04:37:43.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:43 smithi122 ceph-mon[27145]: Reconfiguring daemon mgr.smithi053.legdmv on smithi053 2024-04-02T04:37:44.556 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:44 smithi053 ceph-mon[24889]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:44.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:44 smithi122 ceph-mon[27145]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:45.630 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:45 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:45.630 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:45 smithi053 ceph-mon[24889]: Reconfiguring alertmanager.smithi053 (dependencies changed)... 2024-04-02T04:37:45.630 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:45 smithi053 ceph-mon[24889]: Reconfiguring daemon alertmanager.smithi053 on smithi053 2024-04-02T04:37:45.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:45 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:45.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:45 smithi122 ceph-mon[27145]: Reconfiguring alertmanager.smithi053 (dependencies changed)... 2024-04-02T04:37:45.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:45 smithi122 ceph-mon[27145]: Reconfiguring daemon alertmanager.smithi053 on smithi053 2024-04-02T04:37:46.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:46 smithi053 ceph-mon[24889]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:46.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:46 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-02T04:37:46.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:46 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:37:46.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:46 smithi122 ceph-mon[27145]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:46.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:46 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-02T04:37:46.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:46 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:37:47.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:47 smithi053 ceph-mon[24889]: Deploying daemon osd.0 on smithi053 2024-04-02T04:37:47.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:47 smithi122 ceph-mon[27145]: Deploying daemon osd.0 on smithi053 2024-04-02T04:37:48.617 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:48 smithi053 ceph-mon[24889]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:48.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:48 smithi122 ceph-mon[27145]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:49.643 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:49 smithi053 ceph-mon[24889]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:49.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:49 smithi122 ceph-mon[27145]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:51.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:51 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:51.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:51 smithi053 ceph-mon[24889]: Reconfiguring crash.smithi053 (monmap changed)... 2024-04-02T04:37:51.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:51 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi053", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-02T04:37:51.559 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:51 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:37:51.559 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:51 smithi053 ceph-mon[24889]: Reconfiguring daemon crash.smithi053 on smithi053 2024-04-02T04:37:51.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:51 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:51.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:51 smithi122 ceph-mon[27145]: Reconfiguring crash.smithi053 (monmap changed)... 2024-04-02T04:37:51.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:51 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi053", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-02T04:37:51.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:51 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:37:51.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:51 smithi122 ceph-mon[27145]: Reconfiguring daemon crash.smithi053 on smithi053 2024-04-02T04:37:52.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:52 smithi053 ceph-mon[24889]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:52.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:52 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:37:52.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:52 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/trash_purge_schedule"}]: dispatch 2024-04-02T04:37:52.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:52 smithi122 ceph-mon[27145]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:52.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:52 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:37:52.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:52 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/trash_purge_schedule"}]: dispatch 2024-04-02T04:37:54.171 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:53 smithi053 ceph-mon[24889]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:54.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:53 smithi122 ceph-mon[27145]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:56.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:56 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:56.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:56 smithi122 ceph-mon[27145]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:56.646 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:56 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:56.646 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:56 smithi053 ceph-mon[24889]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:57.763 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:57 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:57.764 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:57 smithi053 ceph-mon[24889]: Reconfiguring grafana.smithi053 (dependencies changed)... 2024-04-02T04:37:57.764 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:57 smithi053 ceph-mon[24889]: Reconfiguring daemon grafana.smithi053 on smithi053 2024-04-02T04:37:57.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:57 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:37:57.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:57 smithi122 ceph-mon[27145]: Reconfiguring grafana.smithi053 (dependencies changed)... 2024-04-02T04:37:57.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:57 smithi122 ceph-mon[27145]: Reconfiguring daemon grafana.smithi053 on smithi053 2024-04-02T04:37:58.696 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:58 smithi122 ceph-mon[27145]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:58.791 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:58 smithi053 ceph-mon[24889]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:37:59.778 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:37:59 smithi053 ceph-mon[24889]: from='osd.0 [v2:172.21.15.53:6802/1162124435,v1:172.21.15.53:6803/1162124435]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-02T04:37:59.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:37:59 smithi122 ceph-mon[27145]: from='osd.0 [v2:172.21.15.53:6802/1162124435,v1:172.21.15.53:6803/1162124435]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-02T04:38:00.800 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:00 smithi053 ceph-mon[24889]: from='osd.0 [v2:172.21.15.53:6802/1162124435,v1:172.21.15.53:6803/1162124435]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-02T04:38:00.800 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:00 smithi053 ceph-mon[24889]: osdmap e7: 1 total, 0 up, 1 in 2024-04-02T04:38:00.800 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:00 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:38:00.801 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:00 smithi053 ceph-mon[24889]: from='osd.0 [v2:172.21.15.53:6802/1162124435,v1:172.21.15.53:6803/1162124435]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-02T04:38:00.801 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:00 smithi053 ceph-mon[24889]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:38:00.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:00 smithi122 ceph-mon[27145]: from='osd.0 [v2:172.21.15.53:6802/1162124435,v1:172.21.15.53:6803/1162124435]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-02T04:38:00.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:00 smithi122 ceph-mon[27145]: osdmap e7: 1 total, 0 up, 1 in 2024-04-02T04:38:00.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:00 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:38:00.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:00 smithi122 ceph-mon[27145]: from='osd.0 [v2:172.21.15.53:6802/1162124435,v1:172.21.15.53:6803/1162124435]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-02T04:38:00.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:00 smithi122 ceph-mon[27145]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:38:01.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:01 smithi053 ceph-mon[24889]: purged_snaps scrub starts 2024-04-02T04:38:01.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:01 smithi053 ceph-mon[24889]: purged_snaps scrub ok 2024-04-02T04:38:01.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:01 smithi053 ceph-mon[24889]: from='osd.0 [v2:172.21.15.53:6802/1162124435,v1:172.21.15.53:6803/1162124435]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-04-02T04:38:01.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:01 smithi053 ceph-mon[24889]: osdmap e8: 1 total, 0 up, 1 in 2024-04-02T04:38:01.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:01 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:38:01.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:01 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:38:01.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:01 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:01.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:01 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi122", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-02T04:38:01.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:01 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:01.634 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:01 smithi122 ceph-mon[27145]: purged_snaps scrub starts 2024-04-02T04:38:01.634 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:01 smithi122 ceph-mon[27145]: purged_snaps scrub ok 2024-04-02T04:38:01.634 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:01 smithi122 ceph-mon[27145]: from='osd.0 [v2:172.21.15.53:6802/1162124435,v1:172.21.15.53:6803/1162124435]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-04-02T04:38:01.634 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:01 smithi122 ceph-mon[27145]: osdmap e8: 1 total, 0 up, 1 in 2024-04-02T04:38:01.634 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:01 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:38:01.634 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:01 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:38:01.635 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:01 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:01.635 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:01 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi122", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-02T04:38:01.635 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:01 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:02.372 INFO:teuthology.orchestra.run.smithi053.stdout:Created osd(s) 0 on host 'smithi053' 2024-04-02T04:38:02.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:02 smithi122 ceph-mon[27145]: Reconfiguring crash.smithi122 (monmap changed)... 2024-04-02T04:38:02.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:02 smithi122 ceph-mon[27145]: Reconfiguring daemon crash.smithi122 on smithi122 2024-04-02T04:38:02.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:02 smithi122 ceph-mon[27145]: osd.0 [v2:172.21.15.53:6802/1162124435,v1:172.21.15.53:6803/1162124435] boot 2024-04-02T04:38:02.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:02 smithi122 ceph-mon[27145]: osdmap e9: 1 total, 1 up, 1 in 2024-04-02T04:38:02.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:02 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:38:02.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:02 smithi122 ceph-mon[27145]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:38:02.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:02 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:02.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:02 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi122.cpwlyk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-02T04:38:02.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:02 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-02T04:38:02.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:02 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:02.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:02 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:02.685 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:02 smithi053 ceph-mon[24889]: Reconfiguring crash.smithi122 (monmap changed)... 2024-04-02T04:38:02.685 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:02 smithi053 ceph-mon[24889]: Reconfiguring daemon crash.smithi122 on smithi122 2024-04-02T04:38:02.685 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:02 smithi053 ceph-mon[24889]: osd.0 [v2:172.21.15.53:6802/1162124435,v1:172.21.15.53:6803/1162124435] boot 2024-04-02T04:38:02.685 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:02 smithi053 ceph-mon[24889]: osdmap e9: 1 total, 1 up, 1 in 2024-04-02T04:38:02.685 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:02 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:38:02.685 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:02 smithi053 ceph-mon[24889]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T04:38:02.685 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:02 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:02.685 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:02 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi122.cpwlyk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-02T04:38:02.685 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:02 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-02T04:38:02.685 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:02 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:02.686 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:02 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:03.027 DEBUG:teuthology.orchestra.run.smithi053:osd.0> sudo journalctl -f -n 0 -u ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.0.service 2024-04-02T04:38:03.030 INFO:tasks.cephadm:Deploying osd.1 on smithi053 with /dev/vg_nvme/lv_3... 2024-04-02T04:38:03.030 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-02T04:38:03.637 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:03 smithi122 ceph-mon[27145]: Reconfiguring mgr.smithi122.cpwlyk (monmap changed)... 2024-04-02T04:38:03.637 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:03 smithi122 ceph-mon[27145]: Reconfiguring daemon mgr.smithi122.cpwlyk on smithi122 2024-04-02T04:38:03.637 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:03 smithi122 ceph-mon[27145]: osdmap e10: 1 total, 1 up, 1 in 2024-04-02T04:38:03.647 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:03 smithi053 ceph-mon[24889]: Reconfiguring mgr.smithi122.cpwlyk (monmap changed)... 2024-04-02T04:38:03.647 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:03 smithi053 ceph-mon[24889]: Reconfiguring daemon mgr.smithi122.cpwlyk on smithi122 2024-04-02T04:38:03.647 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:03 smithi053 ceph-mon[24889]: osdmap e10: 1 total, 1 up, 1 in 2024-04-02T04:38:04.584 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:04 smithi053 ceph-mon[24889]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:04.584 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:04 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:04.584 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:04 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T04:38:04.584 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:04 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-02T04:38:04.584 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:04 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:04.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:04 smithi122 ceph-mon[27145]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:04.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:04 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:04.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:04 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T04:38:04.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:04 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-02T04:38:04.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:04 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:05.057 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-02T04:38:05.057 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-02T04:38:05.057 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10+0 records in 2024-04-02T04:38:05.057 INFO:teuthology.orchestra.run.smithi053.stderr:10+0 records out 2024-04-02T04:38:05.057 INFO:teuthology.orchestra.run.smithi053.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0085522 s, 1.2 GB/s 2024-04-02T04:38:05.057 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping successful for: 2024-04-02T04:38:05.691 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:05 smithi053 ceph-mon[24889]: Reconfiguring mon.smithi122 (monmap changed)... 2024-04-02T04:38:05.691 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:05 smithi053 ceph-mon[24889]: Reconfiguring daemon mon.smithi122 on smithi122 2024-04-02T04:38:05.691 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:05 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:05.691 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:05 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-02T04:38:05.691 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:05 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.53:9093"}]: dispatch 2024-04-02T04:38:05.691 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:05 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:05.691 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:05 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-02T04:38:05.691 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:05 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.53:3000"}]: dispatch 2024-04-02T04:38:05.692 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:05 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:05.692 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:05 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-02T04:38:05.692 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:05 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.53:9095"}]: dispatch 2024-04-02T04:38:05.692 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:05 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:05.692 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:05 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:38:05.692 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:05 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:05.692 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:05 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:38:05.693 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi053:vg_nvme/lv_3 2024-04-02T04:38:05.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:05 smithi122 ceph-mon[27145]: Reconfiguring mon.smithi122 (monmap changed)... 2024-04-02T04:38:05.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:05 smithi122 ceph-mon[27145]: Reconfiguring daemon mon.smithi122 on smithi122 2024-04-02T04:38:05.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:05 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:05.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:05 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-02T04:38:05.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:05 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.53:9093"}]: dispatch 2024-04-02T04:38:05.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:05 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:05.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:05 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-02T04:38:05.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:05 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.53:3000"}]: dispatch 2024-04-02T04:38:05.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:05 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:05.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:05 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-02T04:38:05.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:05 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.53:9095"}]: dispatch 2024-04-02T04:38:05.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:05 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:05.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:05 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:38:05.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:05 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:05.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:05 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:38:06.739 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:06 smithi053 ceph-mon[24889]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-02T04:38:06.740 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:06 smithi053 ceph-mon[24889]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.53:9093"}]: dispatch 2024-04-02T04:38:06.740 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:06 smithi053 ceph-mon[24889]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-02T04:38:06.740 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:06 smithi053 ceph-mon[24889]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.53:3000"}]: dispatch 2024-04-02T04:38:06.740 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:06 smithi053 ceph-mon[24889]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-02T04:38:06.740 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:06 smithi053 ceph-mon[24889]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.53:9095"}]: dispatch 2024-04-02T04:38:06.740 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:06 smithi053 ceph-mon[24889]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:06.890 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:06 smithi122 ceph-mon[27145]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-02T04:38:06.890 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:06 smithi122 ceph-mon[27145]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.53:9093"}]: dispatch 2024-04-02T04:38:06.890 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:06 smithi122 ceph-mon[27145]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-02T04:38:06.890 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:06 smithi122 ceph-mon[27145]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.53:3000"}]: dispatch 2024-04-02T04:38:06.890 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:06 smithi122 ceph-mon[27145]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-02T04:38:06.890 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:06 smithi122 ceph-mon[27145]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.53:9095"}]: dispatch 2024-04-02T04:38:06.890 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:06 smithi122 ceph-mon[27145]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:07.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:07 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:07.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:07 smithi122 ceph-mon[27145]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:07.929 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:07 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:07.929 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:07 smithi053 ceph-mon[24889]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:09.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:08 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:38:09.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:08 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:38:09.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:08 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:09.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:08 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:38:09.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:08 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:38:09.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:08 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:10.127 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:09 smithi053 ceph-mon[24889]: from='client.14258 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:38:10.128 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:09 smithi053 ceph-mon[24889]: pgmap v37: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:10.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:09 smithi122 ceph-mon[27145]: from='client.14258 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:38:10.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:09 smithi122 ceph-mon[27145]: pgmap v37: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:11.750 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:11 smithi053 ceph-mon[24889]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:11.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:11 smithi122 ceph-mon[27145]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:13.045 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:12 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:13.046 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:12 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/1828086793' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ca259ef-434c-4f49-8daa-c00ad50398c2"}]: dispatch 2024-04-02T04:38:13.046 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:12 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/1828086793' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1ca259ef-434c-4f49-8daa-c00ad50398c2"}]': finished 2024-04-02T04:38:13.046 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:12 smithi122 ceph-mon[27145]: osdmap e11: 2 total, 1 up, 2 in 2024-04-02T04:38:13.046 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:12 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:38:13.046 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:12 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/2367373754' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:38:13.056 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:12 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:13.056 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:12 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/1828086793' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ca259ef-434c-4f49-8daa-c00ad50398c2"}]: dispatch 2024-04-02T04:38:13.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:12 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/1828086793' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1ca259ef-434c-4f49-8daa-c00ad50398c2"}]': finished 2024-04-02T04:38:13.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:12 smithi053 ceph-mon[24889]: osdmap e11: 2 total, 1 up, 2 in 2024-04-02T04:38:13.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:12 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:38:13.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:12 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/2367373754' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:38:14.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:13 smithi053 ceph-mon[24889]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:14.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:13 smithi122 ceph-mon[27145]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:15.685 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:15 smithi053 ceph-mon[24889]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:15.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:15 smithi122 ceph-mon[27145]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:17.689 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:17 smithi053 ceph-mon[24889]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:17.827 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:17 smithi122 ceph-mon[27145]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:18.985 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:18 smithi122 ceph-mon[27145]: Detected new or changed devices on smithi053 2024-04-02T04:38:18.985 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:18 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:18.985 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:18 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:38:18.985 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:18 smithi122 ceph-mon[27145]: Adjusting osd_memory_target on smithi053 to 12869M 2024-04-02T04:38:18.986 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:18 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:18.986 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:18 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:18.986 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:18 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:38:18.986 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:18 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:18.986 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:18 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:38:19.025 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:18 smithi053 ceph-mon[24889]: Detected new or changed devices on smithi053 2024-04-02T04:38:19.025 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:18 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:19.025 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:18 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:38:19.025 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:18 smithi053 ceph-mon[24889]: Adjusting osd_memory_target on smithi053 to 12869M 2024-04-02T04:38:19.025 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:18 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:19.025 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:18 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:19.025 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:18 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:38:19.026 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:18 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:19.026 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:18 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:38:19.911 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:19 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-02T04:38:19.911 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:19 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:19.912 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:19 smithi053 ceph-mon[24889]: Deploying daemon osd.1 on smithi053 2024-04-02T04:38:19.912 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:19 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:19.912 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:19 smithi053 ceph-mon[24889]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:20.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:19 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-02T04:38:20.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:19 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:20.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:19 smithi122 ceph-mon[27145]: Deploying daemon osd.1 on smithi053 2024-04-02T04:38:20.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:19 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:20.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:19 smithi122 ceph-mon[27145]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:21.622 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:21 smithi053 ceph-mon[24889]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:21.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:21 smithi122 ceph-mon[27145]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:24.185 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:23 smithi053 ceph-mon[24889]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:24.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:23 smithi122 ceph-mon[27145]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:25.789 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:25 smithi053 ceph-mon[24889]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:25.790 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:25 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:25.790 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:25 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:25.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:25 smithi122 ceph-mon[27145]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:25.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:25 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:25.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:25 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:27.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:27 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:27.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:27 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:38:27.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:27 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:27.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:27 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:38:27.341 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:27 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:27.341 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:27 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:38:27.341 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:27 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:27.341 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:27 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:38:28.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:28 smithi053 ceph-mon[24889]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:28.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:28 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:28.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:28 smithi122 ceph-mon[27145]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:28.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:28 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:29.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:29 smithi053 ceph-mon[24889]: from='osd.1 [v2:172.21.15.53:6810/2808409208,v1:172.21.15.53:6811/2808409208]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-02T04:38:29.605 INFO:teuthology.orchestra.run.smithi053.stdout:Created osd(s) 1 on host 'smithi053' 2024-04-02T04:38:29.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:29 smithi122 ceph-mon[27145]: from='osd.1 [v2:172.21.15.53:6810/2808409208,v1:172.21.15.53:6811/2808409208]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-02T04:38:30.209 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:30 smithi053 ceph-mon[24889]: from='osd.1 [v2:172.21.15.53:6810/2808409208,v1:172.21.15.53:6811/2808409208]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-02T04:38:30.210 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:30 smithi053 ceph-mon[24889]: osdmap e12: 2 total, 1 up, 2 in 2024-04-02T04:38:30.210 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:30 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:38:30.210 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:30 smithi053 ceph-mon[24889]: from='osd.1 [v2:172.21.15.53:6810/2808409208,v1:172.21.15.53:6811/2808409208]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-02T04:38:30.210 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:30 smithi053 ceph-mon[24889]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:30.210 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:30 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:30.233 DEBUG:teuthology.orchestra.run.smithi053:osd.1> sudo journalctl -f -n 0 -u ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.1.service 2024-04-02T04:38:30.235 INFO:tasks.cephadm:Deploying osd.2 on smithi053 with /dev/vg_nvme/lv_2... 2024-04-02T04:38:30.235 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-02T04:38:30.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:30 smithi122 ceph-mon[27145]: from='osd.1 [v2:172.21.15.53:6810/2808409208,v1:172.21.15.53:6811/2808409208]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-02T04:38:30.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:30 smithi122 ceph-mon[27145]: osdmap e12: 2 total, 1 up, 2 in 2024-04-02T04:38:30.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:30 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:38:30.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:30 smithi122 ceph-mon[27145]: from='osd.1 [v2:172.21.15.53:6810/2808409208,v1:172.21.15.53:6811/2808409208]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-02T04:38:30.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:30 smithi122 ceph-mon[27145]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:30.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:30 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:31.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:31 smithi053 ceph-mon[24889]: from='osd.1 [v2:172.21.15.53:6810/2808409208,v1:172.21.15.53:6811/2808409208]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-04-02T04:38:31.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:31 smithi053 ceph-mon[24889]: osdmap e13: 2 total, 1 up, 2 in 2024-04-02T04:38:31.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:31 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:38:31.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:31 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:38:31.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:31 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:38:31.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:31 smithi122 ceph-mon[27145]: from='osd.1 [v2:172.21.15.53:6810/2808409208,v1:172.21.15.53:6811/2808409208]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-04-02T04:38:31.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:31 smithi122 ceph-mon[27145]: osdmap e13: 2 total, 1 up, 2 in 2024-04-02T04:38:31.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:31 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:38:31.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:31 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:38:31.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:31 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:38:32.210 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:32 smithi053 ceph-mon[24889]: purged_snaps scrub starts 2024-04-02T04:38:32.210 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:32 smithi053 ceph-mon[24889]: purged_snaps scrub ok 2024-04-02T04:38:32.210 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:32 smithi053 ceph-mon[24889]: osd.1 [v2:172.21.15.53:6810/2808409208,v1:172.21.15.53:6811/2808409208] boot 2024-04-02T04:38:32.210 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:32 smithi053 ceph-mon[24889]: osdmap e14: 2 total, 2 up, 2 in 2024-04-02T04:38:32.210 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:32 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:38:32.210 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:32 smithi053 ceph-mon[24889]: pgmap v52: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:32.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:32 smithi122 ceph-mon[27145]: purged_snaps scrub starts 2024-04-02T04:38:32.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:32 smithi122 ceph-mon[27145]: purged_snaps scrub ok 2024-04-02T04:38:32.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:32 smithi122 ceph-mon[27145]: osd.1 [v2:172.21.15.53:6810/2808409208,v1:172.21.15.53:6811/2808409208] boot 2024-04-02T04:38:32.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:32 smithi122 ceph-mon[27145]: osdmap e14: 2 total, 2 up, 2 in 2024-04-02T04:38:32.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:32 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:38:32.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:32 smithi122 ceph-mon[27145]: pgmap v52: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-02T04:38:32.696 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-02T04:38:32.696 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-02T04:38:32.696 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10+0 records in 2024-04-02T04:38:32.697 INFO:teuthology.orchestra.run.smithi053.stderr:10+0 records out 2024-04-02T04:38:32.697 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00940689 s, 1.1 GB/s 2024-04-02T04:38:32.697 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping successful for: 2024-04-02T04:38:33.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:33 smithi053 ceph-mon[24889]: osdmap e15: 2 total, 2 up, 2 in 2024-04-02T04:38:33.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:33 smithi122 ceph-mon[27145]: osdmap e15: 2 total, 2 up, 2 in 2024-04-02T04:38:33.688 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi053:vg_nvme/lv_2 2024-04-02T04:38:34.306 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:34 smithi053 ceph-mon[24889]: pgmap v54: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:34.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:34 smithi122 ceph-mon[27145]: pgmap v54: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:35.761 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:35 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:35.761 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:35 smithi053 ceph-mon[24889]: pgmap v55: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:35.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:35 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:35.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:35 smithi122 ceph-mon[27145]: pgmap v55: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:36.726 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:36 smithi053 ceph-mon[24889]: from='client.14276 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:38:36.726 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:36 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:38:36.726 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:36 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:38:36.726 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:36 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:36.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:36 smithi122 ceph-mon[27145]: from='client.14276 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:38:36.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:36 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:38:36.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:36 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:38:36.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:36 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:37.772 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:37 smithi053 ceph-mon[24889]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:37.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:37 smithi122 ceph-mon[27145]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:40.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:39 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:40.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:39 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:38:40.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:39 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:38:40.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:39 smithi122 ceph-mon[27145]: Adjusting osd_memory_target on smithi053 to 6434M 2024-04-02T04:38:40.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:39 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:40.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:39 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:40.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:39 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:38:40.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:39 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:40.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:39 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:38:40.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:39 smithi122 ceph-mon[27145]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:40.226 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:39 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:40.226 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:39 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:38:40.226 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:39 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:38:40.226 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:39 smithi053 ceph-mon[24889]: Adjusting osd_memory_target on smithi053 to 6434M 2024-04-02T04:38:40.226 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:39 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:40.226 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:39 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:40.226 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:39 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:38:40.226 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:39 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:40.226 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:39 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:38:40.226 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:39 smithi053 ceph-mon[24889]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:41.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:40 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/2273906209' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab3c0752-d41b-4164-9335-5aced865ae2a"}]: dispatch 2024-04-02T04:38:41.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:40 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/2273906209' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ab3c0752-d41b-4164-9335-5aced865ae2a"}]': finished 2024-04-02T04:38:41.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:40 smithi122 ceph-mon[27145]: osdmap e16: 3 total, 2 up, 3 in 2024-04-02T04:38:41.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:40 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:41.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:40 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/2273906209' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab3c0752-d41b-4164-9335-5aced865ae2a"}]: dispatch 2024-04-02T04:38:41.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:40 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/2273906209' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ab3c0752-d41b-4164-9335-5aced865ae2a"}]': finished 2024-04-02T04:38:41.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:40 smithi053 ceph-mon[24889]: osdmap e16: 3 total, 2 up, 3 in 2024-04-02T04:38:41.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:40 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:42.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:41 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/663110744' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:38:42.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:41 smithi053 ceph-mon[24889]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:42.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:41 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/663110744' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:38:42.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:41 smithi122 ceph-mon[27145]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:44.106 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:43 smithi053 ceph-mon[24889]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:44.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:43 smithi122 ceph-mon[27145]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:45.437 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:45 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:45.438 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:45 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:45.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:45 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:45.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:45 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:46.493 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:46 smithi053 ceph-mon[24889]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:46.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:46 smithi122 ceph-mon[27145]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:47.456 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:47 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-02T04:38:47.456 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:47 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:47.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:47 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-02T04:38:47.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:47 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:48.402 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:48 smithi053 ceph-mon[24889]: Deploying daemon osd.2 on smithi053 2024-04-02T04:38:48.403 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:48 smithi053 ceph-mon[24889]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:48.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:48 smithi122 ceph-mon[27145]: Deploying daemon osd.2 on smithi053 2024-04-02T04:38:48.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:48 smithi122 ceph-mon[27145]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:49.777 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:49 smithi053 ceph-mon[24889]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:49.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:49 smithi122 ceph-mon[27145]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:51.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:51 smithi053 ceph-mon[24889]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:51.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:51 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:38:51.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:51 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/trash_purge_schedule"}]: dispatch 2024-04-02T04:38:51.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:51 smithi122 ceph-mon[27145]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:51.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:51 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:38:51.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:51 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/trash_purge_schedule"}]: dispatch 2024-04-02T04:38:53.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:52 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:53.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:52 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:38:53.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:52 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:53.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:52 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:38:53.225 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:52 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:53.225 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:52 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:38:53.226 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:52 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:38:53.226 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:52 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:38:54.306 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:53 smithi053 ceph-mon[24889]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:54.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:53 smithi122 ceph-mon[27145]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:54.974 INFO:teuthology.orchestra.run.smithi053.stdout:Created osd(s) 2 on host 'smithi053' 2024-04-02T04:38:55.284 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:55 smithi053 ceph-mon[24889]: from='osd.2 [v2:172.21.15.53:6818/2977813099,v1:172.21.15.53:6819/2977813099]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-02T04:38:55.284 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:55 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:55.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:55 smithi122 ceph-mon[27145]: from='osd.2 [v2:172.21.15.53:6818/2977813099,v1:172.21.15.53:6819/2977813099]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-02T04:38:55.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:55 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:38:55.585 DEBUG:teuthology.orchestra.run.smithi053:osd.2> sudo journalctl -f -n 0 -u ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.2.service 2024-04-02T04:38:55.587 INFO:tasks.cephadm:Deploying osd.3 on smithi053 with /dev/vg_nvme/lv_1... 2024-04-02T04:38:55.587 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-02T04:38:56.149 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:38:56 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2[43072]: 2024-04-02T04:38:56.010+0000 7f0c66590700 -1 osd.2 0 waiting for initial osdmap 2024-04-02T04:38:56.149 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:38:56 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2[43072]: 2024-04-02T04:38:56.013+0000 7f0c6272a700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T04:38:56.149 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:56 smithi053 ceph-mon[24889]: from='osd.2 [v2:172.21.15.53:6818/2977813099,v1:172.21.15.53:6819/2977813099]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-02T04:38:56.149 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:56 smithi053 ceph-mon[24889]: osdmap e17: 3 total, 2 up, 3 in 2024-04-02T04:38:56.149 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:56 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:56.149 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:56 smithi053 ceph-mon[24889]: from='osd.2 [v2:172.21.15.53:6818/2977813099,v1:172.21.15.53:6819/2977813099]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-02T04:38:56.149 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:56 smithi053 ceph-mon[24889]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:56.149 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:56 smithi053 ceph-mon[24889]: from='osd.2 [v2:172.21.15.53:6818/2977813099,v1:172.21.15.53:6819/2977813099]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-04-02T04:38:56.149 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:56 smithi053 ceph-mon[24889]: osdmap e18: 3 total, 2 up, 3 in 2024-04-02T04:38:56.150 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:56 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:56.150 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:56 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:56.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:56 smithi122 ceph-mon[27145]: from='osd.2 [v2:172.21.15.53:6818/2977813099,v1:172.21.15.53:6819/2977813099]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-02T04:38:56.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:56 smithi122 ceph-mon[27145]: osdmap e17: 3 total, 2 up, 3 in 2024-04-02T04:38:56.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:56 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:56.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:56 smithi122 ceph-mon[27145]: from='osd.2 [v2:172.21.15.53:6818/2977813099,v1:172.21.15.53:6819/2977813099]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-02T04:38:56.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:56 smithi122 ceph-mon[27145]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:56.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:56 smithi122 ceph-mon[27145]: from='osd.2 [v2:172.21.15.53:6818/2977813099,v1:172.21.15.53:6819/2977813099]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-04-02T04:38:56.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:56 smithi122 ceph-mon[27145]: osdmap e18: 3 total, 2 up, 3 in 2024-04-02T04:38:56.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:56 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:56.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:56 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:57.295 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:57 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:57.295 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:57 smithi053 ceph-mon[24889]: osd.2 [v2:172.21.15.53:6818/2977813099,v1:172.21.15.53:6819/2977813099] boot 2024-04-02T04:38:57.295 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:57 smithi053 ceph-mon[24889]: osdmap e19: 3 total, 3 up, 3 in 2024-04-02T04:38:57.295 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:57 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:57.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:57 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:57.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:57 smithi122 ceph-mon[27145]: osd.2 [v2:172.21.15.53:6818/2977813099,v1:172.21.15.53:6819/2977813099] boot 2024-04-02T04:38:57.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:57 smithi122 ceph-mon[27145]: osdmap e19: 3 total, 3 up, 3 in 2024-04-02T04:38:57.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:57 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:38:57.995 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-02T04:38:57.995 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-02T04:38:57.995 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10+0 records in 2024-04-02T04:38:57.995 INFO:teuthology.orchestra.run.smithi053.stderr:10+0 records out 2024-04-02T04:38:57.995 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00934083 s, 1.1 GB/s 2024-04-02T04:38:57.995 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping successful for: 2024-04-02T04:38:58.135 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:58 smithi053 ceph-mon[24889]: purged_snaps scrub starts 2024-04-02T04:38:58.135 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:58 smithi053 ceph-mon[24889]: purged_snaps scrub ok 2024-04-02T04:38:58.135 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:58 smithi053 ceph-mon[24889]: pgmap v70: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:58.135 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:58 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-02T04:38:58.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:58 smithi122 ceph-mon[27145]: purged_snaps scrub starts 2024-04-02T04:38:58.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:58 smithi122 ceph-mon[27145]: purged_snaps scrub ok 2024-04-02T04:38:58.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:58 smithi122 ceph-mon[27145]: pgmap v70: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2024-04-02T04:38:58.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:58 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-02T04:38:59.050 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi053:vg_nvme/lv_1 2024-04-02T04:38:59.306 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:59 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-02T04:38:59.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:59 smithi053 ceph-mon[24889]: osdmap e20: 3 total, 3 up, 3 in 2024-04-02T04:38:59.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:38:59 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:38:59.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:59 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-02T04:38:59.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:59 smithi122 ceph-mon[27145]: osdmap e20: 3 total, 3 up, 3 in 2024-04-02T04:38:59.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:38:59 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-02T04:38:59.807 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:38:59 smithi053 sudo[44320]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-02T04:38:59.807 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:38:59 smithi053 sudo[44320]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-02T04:38:59.807 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:38:59 smithi053 sudo[44320]: pam_unix(sudo:session): session closed for user root 2024-04-02T04:39:00.114 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:00 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-02T04:39:00.115 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:00 smithi053 ceph-mon[24889]: osdmap e21: 3 total, 3 up, 3 in 2024-04-02T04:39:00.115 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:00 smithi053 ceph-mon[24889]: pgmap v73: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:00.115 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:00 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:00.115 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:38:59 smithi053 sudo[44357]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-02T04:39:00.115 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:38:59 smithi053 sudo[44357]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-02T04:39:00.115 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:38:59 smithi053 sudo[44357]: pam_unix(sudo:session): session closed for user root 2024-04-02T04:39:00.379 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:39:00 smithi053 sudo[44379]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-02T04:39:00.379 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:39:00 smithi053 sudo[44379]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-02T04:39:00.379 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:39:00 smithi053 sudo[44379]: pam_unix(sudo:session): session closed for user root 2024-04-02T04:39:00.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:00 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-02T04:39:00.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:00 smithi122 ceph-mon[27145]: osdmap e21: 3 total, 3 up, 3 in 2024-04-02T04:39:00.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:00 smithi122 ceph-mon[27145]: pgmap v73: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:00.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:00 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:00.633 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:39:00 smithi053 sudo[44398]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-02T04:39:00.634 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:39:00 smithi053 sudo[44398]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-02T04:39:00.634 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:39:00 smithi053 sudo[44398]: pam_unix(sudo:session): session closed for user root 2024-04-02T04:39:00.902 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:39:00 smithi053 sudo[44403]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-02T04:39:00.902 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:39:00 smithi053 sudo[44403]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-02T04:39:00.903 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:39:00 smithi053 sudo[44403]: pam_unix(sudo:session): session closed for user root 2024-04-02T04:39:01.161 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:39:00 smithi053 sudo[44434]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-02T04:39:01.161 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:39:00 smithi053 sudo[44434]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-02T04:39:01.161 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:39:00 smithi053 sudo[44434]: pam_unix(sudo:session): session closed for user root 2024-04-02T04:39:01.431 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:01 smithi053 sudo[44437]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-02T04:39:01.431 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:01 smithi053 sudo[44437]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-02T04:39:01.431 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:01 smithi053 sudo[44437]: pam_unix(sudo:session): session closed for user root 2024-04-02T04:39:01.687 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:01 smithi122 ceph-mon[27145]: mgrmap e19: smithi053.legdmv(active, since 2m), standbys: smithi122.cpwlyk 2024-04-02T04:39:01.687 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:01 smithi122 ceph-mon[27145]: osdmap e22: 3 total, 3 up, 3 in 2024-04-02T04:39:01.687 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:01 smithi122 ceph-mon[27145]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-02T04:39:01.687 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:01 smithi122 sudo[29472]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-02T04:39:01.700 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:01 smithi053 sudo[44457]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-02T04:39:01.700 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:01 smithi053 sudo[44457]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-02T04:39:01.700 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:01 smithi053 sudo[44457]: pam_unix(sudo:session): session closed for user root 2024-04-02T04:39:01.700 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:01 smithi053 ceph-mon[24889]: mgrmap e19: smithi053.legdmv(active, since 2m), standbys: smithi122.cpwlyk 2024-04-02T04:39:01.701 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:01 smithi053 ceph-mon[24889]: osdmap e22: 3 total, 3 up, 3 in 2024-04-02T04:39:01.701 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:01 smithi053 ceph-mon[24889]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-02T04:39:01.941 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:01 smithi122 sudo[29472]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-02T04:39:01.941 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:01 smithi122 sudo[29472]: pam_unix(sudo:session): session closed for user root 2024-04-02T04:39:02.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:01 smithi122 sudo[29475]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-02T04:39:02.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:01 smithi122 sudo[29475]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-02T04:39:02.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:01 smithi122 sudo[29475]: pam_unix(sudo:session): session closed for user root 2024-04-02T04:39:02.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:02 smithi053 ceph-mon[24889]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-02T04:39:02.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:02 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi053"}]: dispatch 2024-04-02T04:39:02.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:02 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-02T04:39:02.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:02 smithi053 ceph-mon[24889]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-02T04:39:02.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:02 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi053"}]: dispatch 2024-04-02T04:39:02.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:02 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-02T04:39:02.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:02 smithi053 ceph-mon[24889]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:02.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:02 smithi053 ceph-mon[24889]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-02T04:39:02.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:02 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:02.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:02 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:02.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:02 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:39:02.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:02 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:39:02.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:02 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:02.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:02 smithi122 ceph-mon[27145]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-02T04:39:02.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:02 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi053"}]: dispatch 2024-04-02T04:39:02.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:02 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-02T04:39:02.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:02 smithi122 ceph-mon[27145]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-02T04:39:02.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:02 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi053"}]: dispatch 2024-04-02T04:39:02.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:02 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-02T04:39:02.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:02 smithi122 ceph-mon[27145]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:02.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:02 smithi122 ceph-mon[27145]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-02T04:39:02.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:02 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:02.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:02 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:02.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:02 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:39:02.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:02 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:39:02.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:02 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:03.515 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:03 smithi053 ceph-mon[24889]: from='client.14300 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:39:03.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:03 smithi122 ceph-mon[27145]: from='client.14300 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:39:04.799 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:04 smithi053 ceph-mon[24889]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:04.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:04 smithi122 ceph-mon[27145]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:05.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:05 smithi053 ceph-mon[24889]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:05.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:05 smithi122 ceph-mon[27145]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:06.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:06 smithi122 ceph-mon[27145]: Detected new or changed devices on smithi053 2024-04-02T04:39:06.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:06 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:06.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:06 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:06.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:06 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:06.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:06 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:06.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:06 smithi122 ceph-mon[27145]: Adjusting osd_memory_target on smithi053 to 4289M 2024-04-02T04:39:06.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:06 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:06.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:06 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:06.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:06 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:39:06.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:06 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:06.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:06 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:39:06.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:06 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:07.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:06 smithi053 ceph-mon[24889]: Detected new or changed devices on smithi053 2024-04-02T04:39:07.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:06 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:07.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:06 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:07.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:06 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:07.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:06 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:07.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:06 smithi053 ceph-mon[24889]: Adjusting osd_memory_target on smithi053 to 4289M 2024-04-02T04:39:07.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:06 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:07.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:06 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:07.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:06 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:39:07.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:06 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:07.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:06 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:39:07.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:06 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:07.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:07 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/2873114438' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1c720f15-3974-4e40-8ce0-deaa392dcce9"}]: dispatch 2024-04-02T04:39:07.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:07 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/2873114438' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1c720f15-3974-4e40-8ce0-deaa392dcce9"}]': finished 2024-04-02T04:39:07.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:07 smithi122 ceph-mon[27145]: osdmap e23: 4 total, 3 up, 4 in 2024-04-02T04:39:07.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:07 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:39:07.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:07 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/2754703013' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:39:07.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:07 smithi122 ceph-mon[27145]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:08.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:07 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/2873114438' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1c720f15-3974-4e40-8ce0-deaa392dcce9"}]: dispatch 2024-04-02T04:39:08.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:07 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/2873114438' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1c720f15-3974-4e40-8ce0-deaa392dcce9"}]': finished 2024-04-02T04:39:08.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:07 smithi053 ceph-mon[24889]: osdmap e23: 4 total, 3 up, 4 in 2024-04-02T04:39:08.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:07 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:39:08.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:07 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/2754703013' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:39:08.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:07 smithi053 ceph-mon[24889]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:09.746 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:09 smithi053 ceph-mon[24889]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:09.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:09 smithi122 ceph-mon[27145]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:11.673 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:11 smithi053 ceph-mon[24889]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:11.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:11 smithi122 ceph-mon[27145]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:13.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:12 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-02T04:39:13.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:12 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:13.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:12 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-02T04:39:13.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:12 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:14.218 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:13 smithi053 ceph-mon[24889]: Deploying daemon osd.3 on smithi053 2024-04-02T04:39:14.218 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:13 smithi053 ceph-mon[24889]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:14.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:13 smithi122 ceph-mon[27145]: Deploying daemon osd.3 on smithi053 2024-04-02T04:39:14.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:13 smithi122 ceph-mon[27145]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:15.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:15 smithi053 ceph-mon[24889]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:15.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:15 smithi122 ceph-mon[27145]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:17.850 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:17 smithi053 ceph-mon[24889]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:17.860 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:17 smithi122 ceph-mon[27145]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:19.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:18 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:19.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:18 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:39:19.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:18 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:19.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:18 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:39:19.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:18 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:19.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:18 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:39:19.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:18 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:19.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:18 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:39:19.990 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:19 smithi053 ceph-mon[24889]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:20.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:19 smithi122 ceph-mon[27145]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:20.854 INFO:teuthology.orchestra.run.smithi053.stdout:Created osd(s) 3 on host 'smithi053' 2024-04-02T04:39:21.600 DEBUG:teuthology.orchestra.run.smithi053:osd.3> sudo journalctl -f -n 0 -u ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.3.service 2024-04-02T04:39:21.602 INFO:tasks.cephadm:Deploying osd.4 on smithi122 with /dev/vg_nvme/lv_4... 2024-04-02T04:39:21.602 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-02T04:39:21.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:21 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:21.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:21 smithi122 ceph-mon[27145]: from='osd.3 [v2:172.21.15.53:6826/3564481581,v1:172.21.15.53:6827/3564481581]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-02T04:39:21.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:21 smithi122 ceph-mon[27145]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:22.306 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:21 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:22.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:21 smithi053 ceph-mon[24889]: from='osd.3 [v2:172.21.15.53:6826/3564481581,v1:172.21.15.53:6827/3564481581]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-02T04:39:22.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:21 smithi053 ceph-mon[24889]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:23.123 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:22 smithi122 ceph-mon[27145]: from='osd.3 [v2:172.21.15.53:6826/3564481581,v1:172.21.15.53:6827/3564481581]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-02T04:39:23.124 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:22 smithi122 ceph-mon[27145]: osdmap e24: 4 total, 3 up, 4 in 2024-04-02T04:39:23.124 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:22 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:39:23.124 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:22 smithi122 ceph-mon[27145]: from='osd.3 [v2:172.21.15.53:6826/3564481581,v1:172.21.15.53:6827/3564481581]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-02T04:39:23.278 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:22 smithi053 ceph-mon[24889]: from='osd.3 [v2:172.21.15.53:6826/3564481581,v1:172.21.15.53:6827/3564481581]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-02T04:39:23.278 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:22 smithi053 ceph-mon[24889]: osdmap e24: 4 total, 3 up, 4 in 2024-04-02T04:39:23.278 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:22 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:39:23.278 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:22 smithi053 ceph-mon[24889]: from='osd.3 [v2:172.21.15.53:6826/3564481581,v1:172.21.15.53:6827/3564481581]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-02T04:39:23.279 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:39:22 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3[46334]: 2024-04-02T04:39:22.859+0000 7fbf1c4d8700 -1 osd.3 0 waiting for initial osdmap 2024-04-02T04:39:23.279 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:39:22 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3[46334]: 2024-04-02T04:39:22.863+0000 7fbf1466a700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T04:39:23.639 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-02T04:39:23.639 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-02T04:39:23.639 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2024-04-02T04:39:23.639 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2024-04-02T04:39:23.639 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0131339 s, 798 MB/s 2024-04-02T04:39:23.639 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2024-04-02T04:39:23.977 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:23 smithi122 ceph-mon[27145]: from='osd.3 [v2:172.21.15.53:6826/3564481581,v1:172.21.15.53:6827/3564481581]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-04-02T04:39:23.977 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:23 smithi122 ceph-mon[27145]: osdmap e25: 4 total, 3 up, 4 in 2024-04-02T04:39:23.977 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:23 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:39:23.977 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:23 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:39:23.977 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:23 smithi122 ceph-mon[27145]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:24.217 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:23 smithi053 ceph-mon[24889]: from='osd.3 [v2:172.21.15.53:6826/3564481581,v1:172.21.15.53:6827/3564481581]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-04-02T04:39:24.217 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:23 smithi053 ceph-mon[24889]: osdmap e25: 4 total, 3 up, 4 in 2024-04-02T04:39:24.217 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:23 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:39:24.217 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:23 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:39:24.217 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:23 smithi053 ceph-mon[24889]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-02T04:39:24.243 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi122:vg_nvme/lv_4 2024-04-02T04:39:25.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:24 smithi122 ceph-mon[27145]: purged_snaps scrub starts 2024-04-02T04:39:25.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:24 smithi122 ceph-mon[27145]: purged_snaps scrub ok 2024-04-02T04:39:25.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:24 smithi122 ceph-mon[27145]: osd.3 [v2:172.21.15.53:6826/3564481581,v1:172.21.15.53:6827/3564481581] boot 2024-04-02T04:39:25.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:24 smithi122 ceph-mon[27145]: osdmap e26: 4 total, 4 up, 4 in 2024-04-02T04:39:25.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:24 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:39:25.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:24 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:25.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:24 smithi053 ceph-mon[24889]: purged_snaps scrub starts 2024-04-02T04:39:25.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:24 smithi053 ceph-mon[24889]: purged_snaps scrub ok 2024-04-02T04:39:25.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:24 smithi053 ceph-mon[24889]: osd.3 [v2:172.21.15.53:6826/3564481581,v1:172.21.15.53:6827/3564481581] boot 2024-04-02T04:39:25.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:24 smithi053 ceph-mon[24889]: osdmap e26: 4 total, 4 up, 4 in 2024-04-02T04:39:25.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:24 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:39:25.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:24 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:26.060 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:25 smithi122 ceph-mon[27145]: osdmap e27: 4 total, 4 up, 4 in 2024-04-02T04:39:26.060 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:25 smithi122 ceph-mon[27145]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-02T04:39:26.176 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:25 smithi053 ceph-mon[24889]: osdmap e27: 4 total, 4 up, 4 in 2024-04-02T04:39:26.177 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:25 smithi053 ceph-mon[24889]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-02T04:39:27.087 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:26 smithi122 ceph-mon[27145]: from='client.24155 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:39:27.087 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:26 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:39:27.087 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:26 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:39:27.087 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:26 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:27.201 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:26 smithi053 ceph-mon[24889]: from='client.24155 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:39:27.202 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:26 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:39:27.202 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:26 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:39:27.202 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:26 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:28.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:27 smithi122 ceph-mon[27145]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-02T04:39:28.306 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:27 smithi053 ceph-mon[24889]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-02T04:39:29.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:28 smithi122 ceph-mon[27145]: Detected new or changed devices on smithi053 2024-04-02T04:39:29.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:28 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:29.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:28 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:29.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:28 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:29.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:28 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:29.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:28 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:29.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:28 smithi122 ceph-mon[27145]: Adjusting osd_memory_target on smithi053 to 3217M 2024-04-02T04:39:29.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:28 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:29.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:28 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:29.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:28 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:39:29.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:28 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:29.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:28 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:39:29.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:28 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:29.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:28 smithi122 ceph-mon[27145]: from='client.? 172.21.15.122:0/1576580197' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57d48c55-b883-4458-824e-8505ee3b7916"}]: dispatch 2024-04-02T04:39:29.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:28 smithi122 ceph-mon[27145]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57d48c55-b883-4458-824e-8505ee3b7916"}]: dispatch 2024-04-02T04:39:29.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:28 smithi122 ceph-mon[27145]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57d48c55-b883-4458-824e-8505ee3b7916"}]': finished 2024-04-02T04:39:29.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:28 smithi122 ceph-mon[27145]: osdmap e28: 5 total, 4 up, 5 in 2024-04-02T04:39:29.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:28 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:39:29.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:28 smithi122 ceph-mon[27145]: from='client.? 172.21.15.122:0/45068596' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:39:29.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:28 smithi053 ceph-mon[24889]: Detected new or changed devices on smithi053 2024-04-02T04:39:29.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:28 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:29.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:28 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:29.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:28 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:29.308 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:28 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:29.308 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:28 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:29.308 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:28 smithi053 ceph-mon[24889]: Adjusting osd_memory_target on smithi053 to 3217M 2024-04-02T04:39:29.308 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:28 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:29.308 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:28 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:29.308 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:28 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:39:29.308 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:28 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:29.308 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:28 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:39:29.308 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:28 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:29.308 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:28 smithi053 ceph-mon[24889]: from='client.? 172.21.15.122:0/1576580197' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57d48c55-b883-4458-824e-8505ee3b7916"}]: dispatch 2024-04-02T04:39:29.308 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:28 smithi053 ceph-mon[24889]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57d48c55-b883-4458-824e-8505ee3b7916"}]: dispatch 2024-04-02T04:39:29.309 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:28 smithi053 ceph-mon[24889]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57d48c55-b883-4458-824e-8505ee3b7916"}]': finished 2024-04-02T04:39:29.309 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:28 smithi053 ceph-mon[24889]: osdmap e28: 5 total, 4 up, 5 in 2024-04-02T04:39:29.309 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:28 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:39:29.309 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:28 smithi053 ceph-mon[24889]: from='client.? 172.21.15.122:0/45068596' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:39:30.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:29 smithi122 ceph-mon[27145]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-02T04:39:30.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:29 smithi053 ceph-mon[24889]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-02T04:39:31.626 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:31 smithi122 ceph-mon[27145]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-02T04:39:31.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:31 smithi053 ceph-mon[24889]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-02T04:39:34.187 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:33 smithi122 ceph-mon[27145]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-02T04:39:34.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:33 smithi053 ceph-mon[24889]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-02T04:39:35.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:34 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-02T04:39:35.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:34 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:35.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:34 smithi122 ceph-mon[27145]: Deploying daemon osd.4 on smithi122 2024-04-02T04:39:35.306 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:34 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-02T04:39:35.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:34 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:35.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:34 smithi053 ceph-mon[24889]: Deploying daemon osd.4 on smithi122 2024-04-02T04:39:36.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:35 smithi122 ceph-mon[27145]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-02T04:39:36.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:35 smithi053 ceph-mon[24889]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-02T04:39:37.821 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:37 smithi122 ceph-mon[27145]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-02T04:39:38.056 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:37 smithi053 ceph-mon[24889]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-02T04:39:40.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:39 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:40.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:39 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:39:40.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:39 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:40.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:39 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:39:40.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:39 smithi053 ceph-mon[24889]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-02T04:39:40.337 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:39 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:40.337 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:39 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:39:40.337 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:39 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:40.337 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:39 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:39:40.337 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:39 smithi122 ceph-mon[27145]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-02T04:39:41.746 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:41 smithi122 ceph-mon[27145]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-02T04:39:42.017 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 4 on host 'smithi122' 2024-04-02T04:39:42.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:41 smithi053 ceph-mon[24889]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-02T04:39:42.622 DEBUG:teuthology.orchestra.run.smithi122:osd.4> sudo journalctl -f -n 0 -u ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.4.service 2024-04-02T04:39:42.624 INFO:tasks.cephadm:Deploying osd.5 on smithi122 with /dev/vg_nvme/lv_3... 2024-04-02T04:39:42.624 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-02T04:39:43.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:43 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:43.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:43 smithi053 ceph-mon[24889]: from='osd.4 [v2:172.21.15.122:6800/2404754587,v1:172.21.15.122:6801/2404754587]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-02T04:39:43.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:43 smithi053 ceph-mon[24889]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-02T04:39:43.392 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:43 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:43.392 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:43 smithi122 ceph-mon[27145]: from='osd.4 [v2:172.21.15.122:6800/2404754587,v1:172.21.15.122:6801/2404754587]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-02T04:39:43.392 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:43 smithi122 ceph-mon[27145]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-02T04:39:44.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:44 smithi053 ceph-mon[24889]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-02T04:39:44.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:44 smithi053 ceph-mon[24889]: osdmap e29: 5 total, 4 up, 5 in 2024-04-02T04:39:44.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:44 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:39:44.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:44 smithi053 ceph-mon[24889]: from='osd.4 [v2:172.21.15.122:6800/2404754587,v1:172.21.15.122:6801/2404754587]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-02T04:39:44.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:44 smithi053 ceph-mon[24889]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-02T04:39:44.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:44 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:44.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:44 smithi053 ceph-mon[24889]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-02T04:39:44.310 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:44 smithi122 ceph-mon[27145]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-02T04:39:44.310 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:44 smithi122 ceph-mon[27145]: osdmap e29: 5 total, 4 up, 5 in 2024-04-02T04:39:44.310 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:44 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:39:44.311 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:44 smithi122 ceph-mon[27145]: from='osd.4 [v2:172.21.15.122:6800/2404754587,v1:172.21.15.122:6801/2404754587]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-02T04:39:44.311 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:44 smithi122 ceph-mon[27145]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-02T04:39:44.311 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:44 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:44.311 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:44 smithi122 ceph-mon[27145]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-02T04:39:44.311 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:39:44 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4[31397]: 2024-04-02T04:39:44.024+0000 7fce551f1700 -1 osd.4 0 waiting for initial osdmap 2024-04-02T04:39:44.311 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:39:44 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4[31397]: 2024-04-02T04:39:44.029+0000 7fce4f387700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T04:39:45.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:45 smithi053 ceph-mon[24889]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-02T04:39:45.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:45 smithi053 ceph-mon[24889]: osdmap e30: 5 total, 4 up, 5 in 2024-04-02T04:39:45.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:45 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:39:45.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:45 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:39:45.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:45 smithi122 ceph-mon[27145]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-02T04:39:45.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:45 smithi122 ceph-mon[27145]: osdmap e30: 5 total, 4 up, 5 in 2024-04-02T04:39:45.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:45 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:39:45.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:45 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:39:46.283 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:46 smithi122 ceph-mon[27145]: purged_snaps scrub starts 2024-04-02T04:39:46.283 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:46 smithi122 ceph-mon[27145]: purged_snaps scrub ok 2024-04-02T04:39:46.283 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:46 smithi122 ceph-mon[27145]: osd.4 [v2:172.21.15.122:6800/2404754587,v1:172.21.15.122:6801/2404754587] boot 2024-04-02T04:39:46.283 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:46 smithi122 ceph-mon[27145]: osdmap e31: 5 total, 5 up, 5 in 2024-04-02T04:39:46.283 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:46 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:39:46.283 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:46 smithi122 ceph-mon[27145]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:39:46.306 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:46 smithi053 ceph-mon[24889]: purged_snaps scrub starts 2024-04-02T04:39:46.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:46 smithi053 ceph-mon[24889]: purged_snaps scrub ok 2024-04-02T04:39:46.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:46 smithi053 ceph-mon[24889]: osd.4 [v2:172.21.15.122:6800/2404754587,v1:172.21.15.122:6801/2404754587] boot 2024-04-02T04:39:46.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:46 smithi053 ceph-mon[24889]: osdmap e31: 5 total, 5 up, 5 in 2024-04-02T04:39:46.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:46 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:39:46.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:46 smithi053 ceph-mon[24889]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:39:46.667 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-02T04:39:46.667 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-02T04:39:46.667 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2024-04-02T04:39:46.667 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2024-04-02T04:39:46.667 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0120338 s, 871 MB/s 2024-04-02T04:39:46.667 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2024-04-02T04:39:47.267 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:47 smithi122 ceph-mon[27145]: osdmap e32: 5 total, 5 up, 5 in 2024-04-02T04:39:47.292 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi122:vg_nvme/lv_3 2024-04-02T04:39:47.306 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:47 smithi053 ceph-mon[24889]: osdmap e32: 5 total, 5 up, 5 in 2024-04-02T04:39:48.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:48 smithi053 ceph-mon[24889]: osdmap e33: 5 total, 5 up, 5 in 2024-04-02T04:39:48.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:48 smithi053 ceph-mon[24889]: pgmap v109: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-02T04:39:48.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:48 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:48.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:48 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:48.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:48 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:48.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:48 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:48.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:48 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:39:48.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:48 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:48.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:48 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:39:48.308 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:48 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:48.308 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:48 smithi053 ceph-mon[24889]: osdmap e34: 5 total, 5 up, 5 in 2024-04-02T04:39:48.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:48 smithi122 ceph-mon[27145]: osdmap e33: 5 total, 5 up, 5 in 2024-04-02T04:39:48.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:48 smithi122 ceph-mon[27145]: pgmap v109: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-02T04:39:48.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:48 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:48.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:48 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:39:48.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:48 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:48.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:48 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:48.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:48 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:39:48.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:48 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:48.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:48 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:39:48.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:48 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:39:48.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:48 smithi122 ceph-mon[27145]: osdmap e34: 5 total, 5 up, 5 in 2024-04-02T04:39:49.168 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:49 smithi122 ceph-mon[27145]: Detected new or changed devices on smithi122 2024-04-02T04:39:49.169 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:49 smithi122 ceph-mon[27145]: Adjusting osd_memory_target on smithi122 to 15936M 2024-04-02T04:39:49.169 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:49 smithi122 ceph-mon[27145]: osdmap e35: 5 total, 5 up, 5 in 2024-04-02T04:39:49.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:49 smithi053 ceph-mon[24889]: Detected new or changed devices on smithi122 2024-04-02T04:39:49.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:49 smithi053 ceph-mon[24889]: Adjusting osd_memory_target on smithi122 to 15936M 2024-04-02T04:39:49.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:49 smithi053 ceph-mon[24889]: osdmap e35: 5 total, 5 up, 5 in 2024-04-02T04:39:50.190 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:50 smithi122 ceph-mon[27145]: pgmap v112: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-02T04:39:50.190 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:50 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:39:50.190 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:50 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:39:50.190 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:50 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:50.306 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:50 smithi053 ceph-mon[24889]: pgmap v112: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-02T04:39:50.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:50 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:39:50.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:50 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:39:50.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:50 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:51.274 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:51 smithi122 ceph-mon[27145]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:39:51.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:51 smithi053 ceph-mon[24889]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:39:52.318 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:52 smithi122 ceph-mon[27145]: pgmap v113: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-04-02T04:39:52.318 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:52 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:39:52.319 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:52 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/trash_purge_schedule"}]: dispatch 2024-04-02T04:39:52.319 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:52 smithi122 ceph-mon[27145]: from='client.? 172.21.15.122:0/3884210456' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7f1d757-75f9-4972-8f57-98a6452ab1a8"}]: dispatch 2024-04-02T04:39:52.319 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:52 smithi122 ceph-mon[27145]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7f1d757-75f9-4972-8f57-98a6452ab1a8"}]: dispatch 2024-04-02T04:39:52.319 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:52 smithi122 ceph-mon[27145]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a7f1d757-75f9-4972-8f57-98a6452ab1a8"}]': finished 2024-04-02T04:39:52.319 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:52 smithi122 ceph-mon[27145]: osdmap e36: 6 total, 5 up, 6 in 2024-04-02T04:39:52.319 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:52 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:39:52.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:52 smithi053 ceph-mon[24889]: pgmap v113: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-04-02T04:39:52.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:52 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:39:52.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:52 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/trash_purge_schedule"}]: dispatch 2024-04-02T04:39:52.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:52 smithi053 ceph-mon[24889]: from='client.? 172.21.15.122:0/3884210456' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7f1d757-75f9-4972-8f57-98a6452ab1a8"}]: dispatch 2024-04-02T04:39:52.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:52 smithi053 ceph-mon[24889]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7f1d757-75f9-4972-8f57-98a6452ab1a8"}]: dispatch 2024-04-02T04:39:52.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:52 smithi053 ceph-mon[24889]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a7f1d757-75f9-4972-8f57-98a6452ab1a8"}]': finished 2024-04-02T04:39:52.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:52 smithi053 ceph-mon[24889]: osdmap e36: 6 total, 5 up, 6 in 2024-04-02T04:39:52.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:52 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:39:53.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:53 smithi122 ceph-mon[27145]: from='client.? 172.21.15.122:0/3287707017' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:39:53.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:53 smithi053 ceph-mon[24889]: from='client.? 172.21.15.122:0/3287707017' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:39:54.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:54 smithi122 ceph-mon[27145]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:39:54.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:54 smithi053 ceph-mon[24889]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:39:55.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:55 smithi122 ceph-mon[27145]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:39:56.056 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:55 smithi053 ceph-mon[24889]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:39:57.872 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:57 smithi122 ceph-mon[27145]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:39:58.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:57 smithi053 ceph-mon[24889]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:39:59.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:58 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-02T04:39:59.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:58 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:59.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:58 smithi122 ceph-mon[27145]: Deploying daemon osd.5 on smithi122 2024-04-02T04:39:59.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:58 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-02T04:39:59.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:58 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:39:59.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:58 smithi053 ceph-mon[24889]: Deploying daemon osd.5 on smithi122 2024-04-02T04:40:00.127 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:39:59 smithi122 ceph-mon[27145]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:40:00.306 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:39:59 smithi053 ceph-mon[24889]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:40:01.092 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:00 smithi122 ceph-mon[27145]: overall HEALTH_OK 2024-04-02T04:40:01.306 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:00 smithi053 ceph-mon[24889]: overall HEALTH_OK 2024-04-02T04:40:02.216 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:01 smithi122 ceph-mon[27145]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:40:02.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:01 smithi053 ceph-mon[24889]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:40:04.397 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:04 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:04.397 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:04 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:40:04.397 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:04 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:04.398 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:04 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:40:04.398 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:04 smithi122 ceph-mon[27145]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:40:04.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:04 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:04.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:04 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:40:04.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:04 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:04.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:04 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:40:04.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:04 smithi053 ceph-mon[24889]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:40:05.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:05 smithi122 ceph-mon[27145]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:40:05.690 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 5 on host 'smithi122' 2024-04-02T04:40:06.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:05 smithi053 ceph-mon[24889]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:40:06.622 DEBUG:teuthology.orchestra.run.smithi122:osd.5> sudo journalctl -f -n 0 -u ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.5.service 2024-04-02T04:40:06.624 INFO:tasks.cephadm:Deploying osd.6 on smithi122 with /dev/vg_nvme/lv_2... 2024-04-02T04:40:06.625 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-02T04:40:06.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:06 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:06.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:06 smithi122 ceph-mon[27145]: from='osd.5 [v2:172.21.15.122:6808/708363020,v1:172.21.15.122:6809/708363020]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-02T04:40:06.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:06 smithi122 ceph-mon[27145]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-02T04:40:07.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:06 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:07.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:06 smithi053 ceph-mon[24889]: from='osd.5 [v2:172.21.15.122:6808/708363020,v1:172.21.15.122:6809/708363020]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-02T04:40:07.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:06 smithi053 ceph-mon[24889]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-02T04:40:08.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:07 smithi053 ceph-mon[24889]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-02T04:40:08.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:07 smithi053 ceph-mon[24889]: osdmap e37: 6 total, 5 up, 6 in 2024-04-02T04:40:08.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:07 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:40:08.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:07 smithi053 ceph-mon[24889]: from='osd.5 [v2:172.21.15.122:6808/708363020,v1:172.21.15.122:6809/708363020]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-02T04:40:08.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:07 smithi053 ceph-mon[24889]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-02T04:40:08.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:07 smithi053 ceph-mon[24889]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:40:08.126 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:07 smithi122 ceph-mon[27145]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-02T04:40:08.126 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:07 smithi122 ceph-mon[27145]: osdmap e37: 6 total, 5 up, 6 in 2024-04-02T04:40:08.126 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:07 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:40:08.126 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:07 smithi122 ceph-mon[27145]: from='osd.5 [v2:172.21.15.122:6808/708363020,v1:172.21.15.122:6809/708363020]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-02T04:40:08.126 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:07 smithi122 ceph-mon[27145]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-02T04:40:08.126 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:07 smithi122 ceph-mon[27145]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-02T04:40:08.127 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:40:07 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[34379]: 2024-04-02T04:40:07.700+0000 7f0e501e7700 -1 osd.5 0 waiting for initial osdmap 2024-04-02T04:40:08.127 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:40:07 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[34379]: 2024-04-02T04:40:07.706+0000 7f0e4b37f700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T04:40:08.962 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:08 smithi122 ceph-mon[27145]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-02T04:40:08.962 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:08 smithi122 ceph-mon[27145]: osdmap e38: 6 total, 5 up, 6 in 2024-04-02T04:40:08.962 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:08 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:40:08.962 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:08 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:40:08.962 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:08 smithi122 ceph-mon[27145]: osd.5 [v2:172.21.15.122:6808/708363020,v1:172.21.15.122:6809/708363020] boot 2024-04-02T04:40:08.963 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:08 smithi122 ceph-mon[27145]: osdmap e39: 6 total, 6 up, 6 in 2024-04-02T04:40:08.963 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:08 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:40:08.963 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:08 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:09.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:08 smithi053 ceph-mon[24889]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-02T04:40:09.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:08 smithi053 ceph-mon[24889]: osdmap e38: 6 total, 5 up, 6 in 2024-04-02T04:40:09.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:08 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:40:09.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:08 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:40:09.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:08 smithi053 ceph-mon[24889]: osd.5 [v2:172.21.15.122:6808/708363020,v1:172.21.15.122:6809/708363020] boot 2024-04-02T04:40:09.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:08 smithi053 ceph-mon[24889]: osdmap e39: 6 total, 6 up, 6 in 2024-04-02T04:40:09.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:08 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:40:09.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:08 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:09.130 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-02T04:40:09.130 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-02T04:40:09.130 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2024-04-02T04:40:09.130 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2024-04-02T04:40:09.130 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0120914 s, 867 MB/s 2024-04-02T04:40:09.130 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2024-04-02T04:40:10.160 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:09 smithi122 ceph-mon[27145]: purged_snaps scrub starts 2024-04-02T04:40:10.160 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:09 smithi122 ceph-mon[27145]: purged_snaps scrub ok 2024-04-02T04:40:10.160 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:09 smithi122 ceph-mon[27145]: osdmap e40: 6 total, 6 up, 6 in 2024-04-02T04:40:10.160 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:09 smithi122 ceph-mon[27145]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:40:10.161 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi122:vg_nvme/lv_2 2024-04-02T04:40:10.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:09 smithi053 ceph-mon[24889]: purged_snaps scrub starts 2024-04-02T04:40:10.369 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:09 smithi053 ceph-mon[24889]: purged_snaps scrub ok 2024-04-02T04:40:10.370 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:09 smithi053 ceph-mon[24889]: osdmap e40: 6 total, 6 up, 6 in 2024-04-02T04:40:10.370 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:09 smithi053 ceph-mon[24889]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:40:11.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:10 smithi122 ceph-mon[27145]: osdmap e41: 6 total, 6 up, 6 in 2024-04-02T04:40:11.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:10 smithi053 ceph-mon[24889]: osdmap e41: 6 total, 6 up, 6 in 2024-04-02T04:40:12.190 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:11 smithi122 ceph-mon[27145]: osdmap e42: 6 total, 6 up, 6 in 2024-04-02T04:40:12.191 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:11 smithi122 ceph-mon[27145]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:40:12.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:11 smithi053 ceph-mon[24889]: osdmap e42: 6 total, 6 up, 6 in 2024-04-02T04:40:12.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:11 smithi053 ceph-mon[24889]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:40:13.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:12 smithi122 ceph-mon[27145]: osdmap e43: 6 total, 6 up, 6 in 2024-04-02T04:40:13.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:12 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:13.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:12 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:40:13.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:12 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:40:13.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:12 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:13.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:12 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:13.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:12 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:40:13.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:12 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:13.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:12 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:40:13.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:12 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:13.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:12 smithi053 ceph-mon[24889]: osdmap e43: 6 total, 6 up, 6 in 2024-04-02T04:40:13.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:12 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:13.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:12 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:40:13.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:12 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:40:13.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:12 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:13.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:12 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:13.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:12 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:40:13.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:12 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:13.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:12 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:40:13.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:12 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:14.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:13 smithi122 ceph-mon[27145]: Detected new or changed devices on smithi122 2024-04-02T04:40:14.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:13 smithi122 ceph-mon[27145]: Adjusting osd_memory_target on smithi122 to 7968M 2024-04-02T04:40:14.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:13 smithi122 ceph-mon[27145]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:40:14.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:13 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:40:14.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:13 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:40:14.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:13 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:14.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:13 smithi122 ceph-mon[27145]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:40:14.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:13 smithi053 ceph-mon[24889]: Detected new or changed devices on smithi122 2024-04-02T04:40:14.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:13 smithi053 ceph-mon[24889]: Adjusting osd_memory_target on smithi122 to 7968M 2024-04-02T04:40:14.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:13 smithi053 ceph-mon[24889]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:40:14.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:13 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:40:14.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:13 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:40:14.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:13 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:14.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:13 smithi053 ceph-mon[24889]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:40:15.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:15 smithi053 ceph-mon[24889]: from='client.? 172.21.15.122:0/2683858282' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22883c68-5a21-46fb-aecc-10e654a8e2ea"}]: dispatch 2024-04-02T04:40:15.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:15 smithi053 ceph-mon[24889]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22883c68-5a21-46fb-aecc-10e654a8e2ea"}]: dispatch 2024-04-02T04:40:15.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:15 smithi053 ceph-mon[24889]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22883c68-5a21-46fb-aecc-10e654a8e2ea"}]': finished 2024-04-02T04:40:15.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:15 smithi053 ceph-mon[24889]: osdmap e44: 7 total, 6 up, 7 in 2024-04-02T04:40:15.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:15 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:40:15.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:15 smithi053 ceph-mon[24889]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:40:15.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:15 smithi122 ceph-mon[27145]: from='client.? 172.21.15.122:0/2683858282' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22883c68-5a21-46fb-aecc-10e654a8e2ea"}]: dispatch 2024-04-02T04:40:15.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:15 smithi122 ceph-mon[27145]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22883c68-5a21-46fb-aecc-10e654a8e2ea"}]: dispatch 2024-04-02T04:40:15.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:15 smithi122 ceph-mon[27145]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22883c68-5a21-46fb-aecc-10e654a8e2ea"}]': finished 2024-04-02T04:40:15.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:15 smithi122 ceph-mon[27145]: osdmap e44: 7 total, 6 up, 7 in 2024-04-02T04:40:15.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:15 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:40:15.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:15 smithi122 ceph-mon[27145]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:40:16.806 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:16 smithi053 ceph-mon[24889]: from='client.? 172.21.15.122:0/2341778846' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:40:16.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:16 smithi122 ceph-mon[27145]: from='client.? 172.21.15.122:0/2341778846' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:40:17.806 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:17 smithi053 ceph-mon[24889]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:40:17.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:17 smithi122 ceph-mon[27145]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:40:19.871 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:19 smithi053 ceph-mon[24889]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:40:19.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:19 smithi122 ceph-mon[27145]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:40:21.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:21 smithi122 ceph-mon[27145]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:40:22.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:21 smithi053 ceph-mon[24889]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:40:22.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:22 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-02T04:40:22.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:22 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:22.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:22 smithi122 ceph-mon[27145]: Deploying daemon osd.6 on smithi122 2024-04-02T04:40:23.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:22 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-02T04:40:23.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:22 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:23.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:22 smithi053 ceph-mon[24889]: Deploying daemon osd.6 on smithi122 2024-04-02T04:40:24.304 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:23 smithi122 ceph-mon[27145]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:40:24.306 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:23 smithi053 ceph-mon[24889]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:40:25.885 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:25 smithi122 ceph-mon[27145]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:40:26.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:25 smithi053 ceph-mon[24889]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:40:27.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:27 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:27.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:27 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:40:27.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:27 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:27.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:27 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:40:27.861 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:27 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:27.861 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:27 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:40:27.861 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:27 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:27.862 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:27 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:40:28.770 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:28 smithi122 ceph-mon[27145]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:40:28.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:28 smithi053 ceph-mon[24889]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:40:29.846 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:29 smithi122 ceph-mon[27145]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:40:30.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:29 smithi053 ceph-mon[24889]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:40:30.489 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 6 on host 'smithi122' 2024-04-02T04:40:30.794 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:30 smithi122 ceph-mon[27145]: from='osd.6 [v2:172.21.15.122:6816/2302023781,v1:172.21.15.122:6817/2302023781]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-02T04:40:30.794 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:30 smithi122 ceph-mon[27145]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-02T04:40:30.794 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:30 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:30.877 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:30 smithi053 ceph-mon[24889]: from='osd.6 [v2:172.21.15.122:6816/2302023781,v1:172.21.15.122:6817/2302023781]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-02T04:40:30.878 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:30 smithi053 ceph-mon[24889]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-02T04:40:30.878 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:30 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:31.116 DEBUG:teuthology.orchestra.run.smithi122:osd.6> sudo journalctl -f -n 0 -u ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.6.service 2024-04-02T04:40:31.119 INFO:tasks.cephadm:Deploying osd.7 on smithi122 with /dev/vg_nvme/lv_1... 2024-04-02T04:40:31.119 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-02T04:40:31.809 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:31 smithi122 ceph-mon[27145]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-02T04:40:31.809 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:31 smithi122 ceph-mon[27145]: osdmap e45: 7 total, 6 up, 7 in 2024-04-02T04:40:31.809 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:31 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:40:31.809 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:31 smithi122 ceph-mon[27145]: from='osd.6 [v2:172.21.15.122:6816/2302023781,v1:172.21.15.122:6817/2302023781]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-02T04:40:31.809 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:31 smithi122 ceph-mon[27145]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-02T04:40:31.809 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:31 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:31.809 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:31 smithi122 ceph-mon[27145]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:40:31.809 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:40:31 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[37374]: 2024-04-02T04:40:31.582+0000 7f76a07e1700 -1 osd.6 0 waiting for initial osdmap 2024-04-02T04:40:31.809 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:40:31 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[37374]: 2024-04-02T04:40:31.588+0000 7f769a176700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T04:40:32.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:31 smithi053 ceph-mon[24889]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-02T04:40:32.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:31 smithi053 ceph-mon[24889]: osdmap e45: 7 total, 6 up, 7 in 2024-04-02T04:40:32.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:31 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:40:32.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:31 smithi053 ceph-mon[24889]: from='osd.6 [v2:172.21.15.122:6816/2302023781,v1:172.21.15.122:6817/2302023781]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-02T04:40:32.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:31 smithi053 ceph-mon[24889]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-02T04:40:32.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:31 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:32.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:31 smithi053 ceph-mon[24889]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-02T04:40:32.820 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:32 smithi122 ceph-mon[27145]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-02T04:40:32.821 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:32 smithi122 ceph-mon[27145]: osdmap e46: 7 total, 6 up, 7 in 2024-04-02T04:40:32.821 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:32 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:40:32.821 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:32 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:40:33.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:32 smithi053 ceph-mon[24889]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-02T04:40:33.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:32 smithi053 ceph-mon[24889]: osdmap e46: 7 total, 6 up, 7 in 2024-04-02T04:40:33.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:32 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:40:33.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:32 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:40:33.119 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-02T04:40:33.119 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-02T04:40:33.119 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2024-04-02T04:40:33.119 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2024-04-02T04:40:33.119 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0134102 s, 782 MB/s 2024-04-02T04:40:33.119 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2024-04-02T04:40:33.695 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:33 smithi122 ceph-mon[27145]: purged_snaps scrub starts 2024-04-02T04:40:33.695 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:33 smithi122 ceph-mon[27145]: purged_snaps scrub ok 2024-04-02T04:40:33.695 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:33 smithi122 ceph-mon[27145]: osd.6 [v2:172.21.15.122:6816/2302023781,v1:172.21.15.122:6817/2302023781] boot 2024-04-02T04:40:33.695 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:33 smithi122 ceph-mon[27145]: osdmap e47: 7 total, 7 up, 7 in 2024-04-02T04:40:33.696 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:33 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:40:33.696 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:33 smithi122 ceph-mon[27145]: osdmap e48: 7 total, 7 up, 7 in 2024-04-02T04:40:33.696 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:33 smithi122 ceph-mon[27145]: pgmap v147: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:40:33.697 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi122:vg_nvme/lv_1 2024-04-02T04:40:34.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:33 smithi053 ceph-mon[24889]: purged_snaps scrub starts 2024-04-02T04:40:34.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:33 smithi053 ceph-mon[24889]: purged_snaps scrub ok 2024-04-02T04:40:34.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:33 smithi053 ceph-mon[24889]: osd.6 [v2:172.21.15.122:6816/2302023781,v1:172.21.15.122:6817/2302023781] boot 2024-04-02T04:40:34.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:33 smithi053 ceph-mon[24889]: osdmap e47: 7 total, 7 up, 7 in 2024-04-02T04:40:34.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:33 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:40:34.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:33 smithi053 ceph-mon[24889]: osdmap e48: 7 total, 7 up, 7 in 2024-04-02T04:40:34.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:33 smithi053 ceph-mon[24889]: pgmap v147: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:40:35.063 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:34 smithi122 ceph-mon[27145]: osdmap e49: 7 total, 7 up, 7 in 2024-04-02T04:40:35.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:34 smithi053 ceph-mon[24889]: osdmap e49: 7 total, 7 up, 7 in 2024-04-02T04:40:36.253 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:35 smithi122 ceph-mon[27145]: osdmap e50: 7 total, 7 up, 7 in 2024-04-02T04:40:36.253 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:35 smithi122 ceph-mon[27145]: pgmap v150: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:40:36.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:35 smithi053 ceph-mon[24889]: osdmap e50: 7 total, 7 up, 7 in 2024-04-02T04:40:36.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:35 smithi053 ceph-mon[24889]: pgmap v150: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:40:37.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:36 smithi122 ceph-mon[27145]: osdmap e51: 7 total, 7 up, 7 in 2024-04-02T04:40:37.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:36 smithi053 ceph-mon[24889]: osdmap e51: 7 total, 7 up, 7 in 2024-04-02T04:40:38.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:38 smithi122 ceph-mon[27145]: Detected new or changed devices on smithi122 2024-04-02T04:40:38.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:38 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:38.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:38 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:40:38.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:38 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:40:38.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:38 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:40:38.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:38 smithi122 ceph-mon[27145]: Adjusting osd_memory_target on smithi122 to 5312M 2024-04-02T04:40:38.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:38 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:38.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:38 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:38.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:38 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:40:38.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:38 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:38.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:38 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:40:38.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:38 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:38.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:38 smithi122 ceph-mon[27145]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:40:38.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:38 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:40:38.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:38 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:40:38.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:38 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:38.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:38 smithi122 ceph-mon[27145]: pgmap v152: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:40:38.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:38 smithi053 ceph-mon[24889]: Detected new or changed devices on smithi122 2024-04-02T04:40:38.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:38 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:38.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:38 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:40:38.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:38 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:40:38.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:38 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:40:38.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:38 smithi053 ceph-mon[24889]: Adjusting osd_memory_target on smithi122 to 5312M 2024-04-02T04:40:38.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:38 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:38.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:38 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:38.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:38 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:40:38.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:38 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:38.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:38 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:40:38.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:38 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:38.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:38 smithi053 ceph-mon[24889]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:40:38.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:38 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T04:40:38.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:38 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T04:40:38.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:38 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:38.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:38 smithi053 ceph-mon[24889]: pgmap v152: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:40:39.866 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:39 smithi122 ceph-mon[27145]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-02T04:40:40.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:39 smithi053 ceph-mon[24889]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-02T04:40:40.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:40 smithi122 ceph-mon[27145]: from='client.? 172.21.15.122:0/1245044369' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db0763d2-09d4-44e9-ae0b-9d6b43d363d2"}]: dispatch 2024-04-02T04:40:40.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:40 smithi122 ceph-mon[27145]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db0763d2-09d4-44e9-ae0b-9d6b43d363d2"}]: dispatch 2024-04-02T04:40:40.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:40 smithi122 ceph-mon[27145]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db0763d2-09d4-44e9-ae0b-9d6b43d363d2"}]': finished 2024-04-02T04:40:40.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:40 smithi122 ceph-mon[27145]: osdmap e52: 8 total, 7 up, 8 in 2024-04-02T04:40:40.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:40 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:40:40.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:40 smithi122 ceph-mon[27145]: from='client.? 172.21.15.122:0/656240441' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:40:41.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:40 smithi053 ceph-mon[24889]: from='client.? 172.21.15.122:0/1245044369' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db0763d2-09d4-44e9-ae0b-9d6b43d363d2"}]: dispatch 2024-04-02T04:40:41.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:40 smithi053 ceph-mon[24889]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db0763d2-09d4-44e9-ae0b-9d6b43d363d2"}]: dispatch 2024-04-02T04:40:41.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:40 smithi053 ceph-mon[24889]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db0763d2-09d4-44e9-ae0b-9d6b43d363d2"}]': finished 2024-04-02T04:40:41.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:40 smithi053 ceph-mon[24889]: osdmap e52: 8 total, 7 up, 8 in 2024-04-02T04:40:41.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:40 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:40:41.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:40 smithi053 ceph-mon[24889]: from='client.? 172.21.15.122:0/656240441' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T04:40:41.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:41 smithi122 ceph-mon[27145]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-02T04:40:42.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:41 smithi053 ceph-mon[24889]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-02T04:40:44.090 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:43 smithi122 ceph-mon[27145]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-02T04:40:44.306 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:43 smithi053 ceph-mon[24889]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-02T04:40:45.808 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:45 smithi122 ceph-mon[27145]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-04-02T04:40:46.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:45 smithi053 ceph-mon[24889]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-04-02T04:40:46.842 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:46 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-02T04:40:46.842 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:46 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:46.842 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:46 smithi053 ceph-mon[24889]: Deploying daemon osd.7 on smithi122 2024-04-02T04:40:46.854 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:46 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-02T04:40:46.854 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:46 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:46.854 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:46 smithi122 ceph-mon[27145]: Deploying daemon osd.7 on smithi122 2024-04-02T04:40:47.809 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:47 smithi122 ceph-mon[27145]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-02T04:40:48.056 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:47 smithi053 ceph-mon[24889]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-02T04:40:49.859 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:49 smithi053 ceph-mon[24889]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:40:49.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:49 smithi122 ceph-mon[27145]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:40:52.509 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:52 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:52.510 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:52 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:40:52.510 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:52 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:52.510 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:52 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:40:52.510 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:52 smithi122 ceph-mon[27145]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:40:52.510 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:52 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:40:52.510 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:52 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/trash_purge_schedule"}]: dispatch 2024-04-02T04:40:52.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:52 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:52.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:52 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:40:52.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:52 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:52.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:52 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:40:52.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:52 smithi053 ceph-mon[24889]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:40:52.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:52 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:40:52.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:52 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/trash_purge_schedule"}]: dispatch 2024-04-02T04:40:54.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:53 smithi122 ceph-mon[27145]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:40:54.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:53 smithi053 ceph-mon[24889]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:40:54.589 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 7 on host 'smithi122' 2024-04-02T04:40:55.229 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:54 smithi122 ceph-mon[27145]: from='osd.7 [v2:172.21.15.122:6824/588529180,v1:172.21.15.122:6825/588529180]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-02T04:40:55.229 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:54 smithi122 ceph-mon[27145]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-02T04:40:55.229 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:54 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:55.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:54 smithi053 ceph-mon[24889]: from='osd.7 [v2:172.21.15.122:6824/588529180,v1:172.21.15.122:6825/588529180]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-02T04:40:55.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:54 smithi053 ceph-mon[24889]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-02T04:40:55.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:54 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:55.527 DEBUG:teuthology.orchestra.run.smithi122:osd.7> sudo journalctl -f -n 0 -u ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.7.service 2024-04-02T04:40:55.530 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-04-02T04:40:55.530 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph osd stat -f json 2024-04-02T04:40:56.142 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:55 smithi053 ceph-mon[24889]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-02T04:40:56.143 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:55 smithi053 ceph-mon[24889]: osdmap e53: 8 total, 7 up, 8 in 2024-04-02T04:40:56.143 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:55 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:40:56.143 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:55 smithi053 ceph-mon[24889]: from='osd.7 [v2:172.21.15.122:6824/588529180,v1:172.21.15.122:6825/588529180]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-02T04:40:56.143 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:55 smithi053 ceph-mon[24889]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-02T04:40:56.143 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:55 smithi053 ceph-mon[24889]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:40:56.143 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:55 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:56.208 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:55 smithi122 ceph-mon[27145]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-02T04:40:56.208 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:55 smithi122 ceph-mon[27145]: osdmap e53: 8 total, 7 up, 8 in 2024-04-02T04:40:56.208 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:55 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:40:56.208 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:55 smithi122 ceph-mon[27145]: from='osd.7 [v2:172.21.15.122:6824/588529180,v1:172.21.15.122:6825/588529180]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-02T04:40:56.208 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:55 smithi122 ceph-mon[27145]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-02T04:40:56.208 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:55 smithi122 ceph-mon[27145]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-02T04:40:56.208 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:55 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:56.208 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:40:55 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[40446]: 2024-04-02T04:40:55.951+0000 7f64d7a4d700 -1 osd.7 0 waiting for initial osdmap 2024-04-02T04:40:56.208 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:40:55 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[40446]: 2024-04-02T04:40:55.961+0000 7f64d23e4700 -1 osd.7 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T04:40:57.133 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:56 smithi053 ceph-mon[24889]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-02T04:40:57.133 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:56 smithi053 ceph-mon[24889]: osdmap e54: 8 total, 7 up, 8 in 2024-04-02T04:40:57.133 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:56 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:40:57.133 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:56 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:40:57.173 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:56 smithi122 ceph-mon[27145]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-02T04:40:57.173 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:56 smithi122 ceph-mon[27145]: osdmap e54: 8 total, 7 up, 8 in 2024-04-02T04:40:57.173 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:56 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:40:57.173 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:56 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:40:57.416 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-02T04:40:58.017 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":55,"num_osds":8,"num_up_osds":8,"osd_up_since":1712032856,"num_in_osds":8,"osd_in_since":1712032839,"num_remapped_pgs":1} 2024-04-02T04:40:58.018 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-02T04:40:58.210 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:57 smithi122 ceph-mon[27145]: purged_snaps scrub starts 2024-04-02T04:40:58.210 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:57 smithi122 ceph-mon[27145]: purged_snaps scrub ok 2024-04-02T04:40:58.210 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:57 smithi122 ceph-mon[27145]: osd.7 [v2:172.21.15.122:6824/588529180,v1:172.21.15.122:6825/588529180] boot 2024-04-02T04:40:58.210 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:57 smithi122 ceph-mon[27145]: osdmap e55: 8 total, 8 up, 8 in 2024-04-02T04:40:58.210 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:57 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:40:58.211 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:57 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/2726414945' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-02T04:40:58.211 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:57 smithi122 ceph-mon[27145]: pgmap v166: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:40:58.211 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:57 smithi122 ceph-mon[27145]: osdmap e56: 8 total, 8 up, 8 in 2024-04-02T04:40:58.242 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/mon.smithi053/config 2024-04-02T04:40:58.280 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:57 smithi053 ceph-mon[24889]: purged_snaps scrub starts 2024-04-02T04:40:58.280 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:57 smithi053 ceph-mon[24889]: purged_snaps scrub ok 2024-04-02T04:40:58.280 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:57 smithi053 ceph-mon[24889]: osd.7 [v2:172.21.15.122:6824/588529180,v1:172.21.15.122:6825/588529180] boot 2024-04-02T04:40:58.281 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:57 smithi053 ceph-mon[24889]: osdmap e55: 8 total, 8 up, 8 in 2024-04-02T04:40:58.281 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:57 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:40:58.281 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:57 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/2726414945' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-02T04:40:58.281 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:57 smithi053 ceph-mon[24889]: pgmap v166: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:40:58.281 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:57 smithi053 ceph-mon[24889]: osdmap e56: 8 total, 8 up, 8 in 2024-04-02T04:40:59.562 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:59 smithi053 ceph-mon[24889]: Detected new or changed devices on smithi122 2024-04-02T04:40:59.562 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:59 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:59.562 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:59 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:40:59.562 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:59 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:40:59.562 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:59 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:40:59.562 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:59 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:40:59.562 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:59 smithi053 ceph-mon[24889]: Adjusting osd_memory_target on smithi122 to 3984M 2024-04-02T04:40:59.562 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:59 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:59.562 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:59 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:59.562 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:59 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:40:59.562 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:59 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:59.562 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:59 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:40:59.563 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:59 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:59.563 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:59 smithi053 ceph-mon[24889]: osdmap e57: 8 total, 8 up, 8 in 2024-04-02T04:40:59.563 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:40:59 smithi053 ceph-mon[24889]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:40:59.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:59 smithi122 ceph-mon[27145]: Detected new or changed devices on smithi122 2024-04-02T04:40:59.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:59 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:59.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:59 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:40:59.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:59 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:40:59.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:59 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:40:59.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:59 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:40:59.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:59 smithi122 ceph-mon[27145]: Adjusting osd_memory_target on smithi122 to 3984M 2024-04-02T04:40:59.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:59 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:59.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:59 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:59.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:59 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:40:59.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:59 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:40:59.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:59 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:40:59.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:59 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:40:59.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:59 smithi122 ceph-mon[27145]: osdmap e57: 8 total, 8 up, 8 in 2024-04-02T04:40:59.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:40:59 smithi122 ceph-mon[27145]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:40:59.960 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-02T04:40:59.960 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":58,"fsid":"48e16742-f0aa-11ee-b647-cb9ed24678a4","created":"2024-04-02T04:34:58.695179+0000","modified":"2024-04-02T04:40:59.936239+0000","last_up_change":"2024-04-02T04:40:56.944274+0000","last_in_change":"2024-04-02T04:40:39.849253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T04:38:57.767014+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"98971ffe-6ad2-4adc-8d2d-7b19f4efc239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1162124435},{"type":"v1","addr":"172.21.15.53:6803","nonce":1162124435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1162124435},{"type":"v1","addr":"172.21.15.53:6805","nonce":1162124435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":1162124435},{"type":"v1","addr":"172.21.15.53:6809","nonce":1162124435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1162124435},{"type":"v1","addr":"172.21.15.53:6807","nonce":1162124435}]},"public_addr":"172.21.15.53:6803/1162124435","cluster_addr":"172.21.15.53:6805/1162124435","heartbeat_back_addr":"172.21.15.53:6809/1162124435","heartbeat_front_addr":"172.21.15.53:6807/1162124435","state":["exists","up"]},{"osd":1,"uuid":"1ca259ef-434c-4f49-8daa-c00ad50398c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2808409208},{"type":"v1","addr":"172.21.15.53:6811","nonce":2808409208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2808409208},{"type":"v1","addr":"172.21.15.53:6813","nonce":2808409208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":2808409208},{"type":"v1","addr":"172.21.15.53:6817","nonce":2808409208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2808409208},{"type":"v1","addr":"172.21.15.53:6815","nonce":2808409208}]},"public_addr":"172.21.15.53:6811/2808409208","cluster_addr":"172.21.15.53:6813/2808409208","heartbeat_back_addr":"172.21.15.53:6817/2808409208","heartbeat_front_addr":"172.21.15.53:6815/2808409208","state":["exists","up"]},{"osd":2,"uuid":"ab3c0752-d41b-4164-9335-5aced865ae2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":2977813099},{"type":"v1","addr":"172.21.15.53:6819","nonce":2977813099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":2977813099},{"type":"v1","addr":"172.21.15.53:6821","nonce":2977813099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2977813099},{"type":"v1","addr":"172.21.15.53:6825","nonce":2977813099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":2977813099},{"type":"v1","addr":"172.21.15.53:6823","nonce":2977813099}]},"public_addr":"172.21.15.53:6819/2977813099","cluster_addr":"172.21.15.53:6821/2977813099","heartbeat_back_addr":"172.21.15.53:6825/2977813099","heartbeat_front_addr":"172.21.15.53:6823/2977813099","state":["exists","up"]},{"osd":3,"uuid":"1c720f15-3974-4e40-8ce0-deaa392dcce9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3564481581},{"type":"v1","addr":"172.21.15.53:6827","nonce":3564481581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3564481581},{"type":"v1","addr":"172.21.15.53:6829","nonce":3564481581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3564481581},{"type":"v1","addr":"172.21.15.53:6833","nonce":3564481581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3564481581},{"type":"v1","addr":"172.21.15.53:6831","nonce":3564481581}]},"public_addr":"172.21.15.53:6827/3564481581","cluster_addr":"172.21.15.53:6829/3564481581","heartbeat_back_addr":"172.21.15.53:6833/3564481581","heartbeat_front_addr":"172.21.15.53:6831/3564481581","state":["exists","up"]},{"osd":4,"uuid":"57d48c55-b883-4458-824e-8505ee3b7916","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2404754587},{"type":"v1","addr":"172.21.15.122:6801","nonce":2404754587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2404754587},{"type":"v1","addr":"172.21.15.122:6803","nonce":2404754587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2404754587},{"type":"v1","addr":"172.21.15.122:6807","nonce":2404754587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2404754587},{"type":"v1","addr":"172.21.15.122:6805","nonce":2404754587}]},"public_addr":"172.21.15.122:6801/2404754587","cluster_addr":"172.21.15.122:6803/2404754587","heartbeat_back_addr":"172.21.15.122:6807/2404754587","heartbeat_front_addr":"172.21.15.122:6805/2404754587","state":["exists","up"]},{"osd":5,"uuid":"a7f1d757-75f9-4972-8f57-98a6452ab1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":708363020},{"type":"v1","addr":"172.21.15.122:6809","nonce":708363020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":708363020},{"type":"v1","addr":"172.21.15.122:6811","nonce":708363020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":708363020},{"type":"v1","addr":"172.21.15.122:6815","nonce":708363020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":708363020},{"type":"v1","addr":"172.21.15.122:6813","nonce":708363020}]},"public_addr":"172.21.15.122:6809/708363020","cluster_addr":"172.21.15.122:6811/708363020","heartbeat_back_addr":"172.21.15.122:6815/708363020","heartbeat_front_addr":"172.21.15.122:6813/708363020","state":["exists","up"]},{"osd":6,"uuid":"22883c68-5a21-46fb-aecc-10e654a8e2ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":2302023781},{"type":"v1","addr":"172.21.15.122:6817","nonce":2302023781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":2302023781},{"type":"v1","addr":"172.21.15.122:6819","nonce":2302023781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":2302023781},{"type":"v1","addr":"172.21.15.122:6823","nonce":2302023781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":2302023781},{"type":"v1","addr":"172.21.15.122:6821","nonce":2302023781}]},"public_addr":"172.21.15.122:6817/2302023781","cluster_addr":"172.21.15.122:6819/2302023781","heartbeat_back_addr":"172.21.15.122:6823/2302023781","heartbeat_front_addr":"172.21.15.122:6821/2302023781","state":["exists","up"]},{"osd":7,"uuid":"db0763d2-09d4-44e9-ae0b-9d6b43d363d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":588529180},{"type":"v1","addr":"172.21.15.122:6825","nonce":588529180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":588529180},{"type":"v1","addr":"172.21.15.122:6827","nonce":588529180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":588529180},{"type":"v1","addr":"172.21.15.122:6831","nonce":588529180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":588529180},{"type":"v1","addr":"172.21.15.122:6829","nonce":588529180}]},"public_addr":"172.21.15.122:6825/588529180","cluster_addr":"172.21.15.122:6827/588529180","heartbeat_back_addr":"172.21.15.122:6831/588529180","heartbeat_front_addr":"172.21.15.122:6829/588529180","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:37:59.442507+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:38:30.071594+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:38:55.953833+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:39:21.900881+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:39:43.074154+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:40:07.139023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:40:30.598928+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:40:55.305914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:0/773205658":"2024-04-03T04:36:51.400435+0000","172.21.15.53:6800/1928933950":"2024-04-03T04:36:00.417623+0000","172.21.15.53:6801/345491441":"2024-04-03T04:36:51.400435+0000","172.21.15.53:0/1377540658":"2024-04-03T04:36:00.417623+0000","172.21.15.53:0/644794640":"2024-04-03T04:36:00.417623+0000","172.21.15.53:0/65784368":"2024-04-03T04:35:31.902665+0000","172.21.15.53:0/225187350":"2024-04-03T04:36:51.400435+0000","172.21.15.53:6801/1928933950":"2024-04-03T04:36:00.417623+0000","172.21.15.53:0/984413801":"2024-04-03T04:35:31.902665+0000","172.21.15.53:6800/345491441":"2024-04-03T04:36:51.400435+0000","172.21.15.53:6800/1387513254":"2024-04-03T04:35:31.902665+0000","172.21.15.53:6801/1387513254":"2024-04-03T04:35:31.902665+0000","172.21.15.53:0/445047527":"2024-04-03T04:36:51.400435+0000","172.21.15.53:0/92223541":"2024-04-03T04:36:00.417623+0000","172.21.15.53:0/1772532444":"2024-04-03T04:35:31.902665+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T04:41:00.610 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T04:38:57.767014+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-04-02T04:41:00.611 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph osd pool get .mgr pg_num 2024-04-02T04:41:00.835 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/mon.smithi053/config 2024-04-02T04:41:01.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:00 smithi053 ceph-mon[24889]: osdmap e58: 8 total, 8 up, 8 in 2024-04-02T04:41:01.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:00 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/3998036281' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T04:41:01.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:00 smithi122 ceph-mon[27145]: osdmap e58: 8 total, 8 up, 8 in 2024-04-02T04:41:01.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:00 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/3998036281' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T04:41:02.202 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:01 smithi053 ceph-mon[24889]: osdmap e59: 8 total, 8 up, 8 in 2024-04-02T04:41:02.202 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:01 smithi053 ceph-mon[24889]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:02.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:01 smithi122 ceph-mon[27145]: osdmap e59: 8 total, 8 up, 8 in 2024-04-02T04:41:02.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:01 smithi122 ceph-mon[27145]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:02.696 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-04-02T04:41:03.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:02 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/1301403401' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T04:41:03.383 INFO:tasks.cephadm:Setting up client nodes... 2024-04-02T04:41:03.384 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-02T04:41:03.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:02 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/1301403401' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T04:41:04.306 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:03 smithi053 ceph-mon[24889]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-02T04:41:04.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:03 smithi122 ceph-mon[27145]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-02T04:41:05.457 INFO:teuthology.orchestra.run.smithi053.stdout:[client.0] 2024-04-02T04:41:05.458 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQBhjAtmeKbZGhAAfUpQNMEnLEcGfbhR2g8T5Q== 2024-04-02T04:41:05.748 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:05 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/3116449225' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-02T04:41:05.748 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:05 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/3116449225' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-02T04:41:05.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:05 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/3116449225' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-02T04:41:05.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:05 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/3116449225' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-02T04:41:06.124 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-02T04:41:06.124 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-02T04:41:06.125 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-02T04:41:06.168 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-02T04:41:06.168 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-02T04:41:06.168 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph mgr dump --format=json 2024-04-02T04:41:06.428 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/mon.smithi053/config 2024-04-02T04:41:06.777 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:06 smithi053 ceph-mon[24889]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-02T04:41:06.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:06 smithi122 ceph-mon[27145]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-02T04:41:08.170 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-02T04:41:08.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:08 smithi053 ceph-mon[24889]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-02T04:41:08.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:08 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/2128480780' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-02T04:41:08.861 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":19,"active_gid":14206,"active_name":"smithi053.legdmv","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":3224076586},{"type":"v1","addr":"172.21.15.53:6801","nonce":3224076586}]},"active_addr":"172.21.15.53:6801/3224076586","active_change":"2024-04-02T04:36:51.400719+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":14226,"name":"smithi122.cpwlyk","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.53:8443/","prometheus":"http://172.21.15.53:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":5,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":3863410258}]},{"addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":1146102633}]},{"addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":3353533888}]},{"addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":2011784789}]}]}} 2024-04-02T04:41:08.868 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-02T04:41:08.868 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-02T04:41:08.868 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-02T04:41:08.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:08 smithi122 ceph-mon[27145]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-02T04:41:08.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:08 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/2128480780' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-02T04:41:09.089 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/mon.smithi053/config 2024-04-02T04:41:10.785 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-02T04:41:10.786 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":59,"fsid":"48e16742-f0aa-11ee-b647-cb9ed24678a4","created":"2024-04-02T04:34:58.695179+0000","modified":"2024-04-02T04:41:00.940645+0000","last_up_change":"2024-04-02T04:40:56.944274+0000","last_in_change":"2024-04-02T04:40:39.849253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T04:38:57.767014+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"98971ffe-6ad2-4adc-8d2d-7b19f4efc239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1162124435},{"type":"v1","addr":"172.21.15.53:6803","nonce":1162124435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1162124435},{"type":"v1","addr":"172.21.15.53:6805","nonce":1162124435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":1162124435},{"type":"v1","addr":"172.21.15.53:6809","nonce":1162124435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1162124435},{"type":"v1","addr":"172.21.15.53:6807","nonce":1162124435}]},"public_addr":"172.21.15.53:6803/1162124435","cluster_addr":"172.21.15.53:6805/1162124435","heartbeat_back_addr":"172.21.15.53:6809/1162124435","heartbeat_front_addr":"172.21.15.53:6807/1162124435","state":["exists","up"]},{"osd":1,"uuid":"1ca259ef-434c-4f49-8daa-c00ad50398c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2808409208},{"type":"v1","addr":"172.21.15.53:6811","nonce":2808409208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2808409208},{"type":"v1","addr":"172.21.15.53:6813","nonce":2808409208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":2808409208},{"type":"v1","addr":"172.21.15.53:6817","nonce":2808409208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2808409208},{"type":"v1","addr":"172.21.15.53:6815","nonce":2808409208}]},"public_addr":"172.21.15.53:6811/2808409208","cluster_addr":"172.21.15.53:6813/2808409208","heartbeat_back_addr":"172.21.15.53:6817/2808409208","heartbeat_front_addr":"172.21.15.53:6815/2808409208","state":["exists","up"]},{"osd":2,"uuid":"ab3c0752-d41b-4164-9335-5aced865ae2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":2977813099},{"type":"v1","addr":"172.21.15.53:6819","nonce":2977813099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":2977813099},{"type":"v1","addr":"172.21.15.53:6821","nonce":2977813099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2977813099},{"type":"v1","addr":"172.21.15.53:6825","nonce":2977813099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":2977813099},{"type":"v1","addr":"172.21.15.53:6823","nonce":2977813099}]},"public_addr":"172.21.15.53:6819/2977813099","cluster_addr":"172.21.15.53:6821/2977813099","heartbeat_back_addr":"172.21.15.53:6825/2977813099","heartbeat_front_addr":"172.21.15.53:6823/2977813099","state":["exists","up"]},{"osd":3,"uuid":"1c720f15-3974-4e40-8ce0-deaa392dcce9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3564481581},{"type":"v1","addr":"172.21.15.53:6827","nonce":3564481581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3564481581},{"type":"v1","addr":"172.21.15.53:6829","nonce":3564481581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3564481581},{"type":"v1","addr":"172.21.15.53:6833","nonce":3564481581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3564481581},{"type":"v1","addr":"172.21.15.53:6831","nonce":3564481581}]},"public_addr":"172.21.15.53:6827/3564481581","cluster_addr":"172.21.15.53:6829/3564481581","heartbeat_back_addr":"172.21.15.53:6833/3564481581","heartbeat_front_addr":"172.21.15.53:6831/3564481581","state":["exists","up"]},{"osd":4,"uuid":"57d48c55-b883-4458-824e-8505ee3b7916","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2404754587},{"type":"v1","addr":"172.21.15.122:6801","nonce":2404754587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2404754587},{"type":"v1","addr":"172.21.15.122:6803","nonce":2404754587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2404754587},{"type":"v1","addr":"172.21.15.122:6807","nonce":2404754587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2404754587},{"type":"v1","addr":"172.21.15.122:6805","nonce":2404754587}]},"public_addr":"172.21.15.122:6801/2404754587","cluster_addr":"172.21.15.122:6803/2404754587","heartbeat_back_addr":"172.21.15.122:6807/2404754587","heartbeat_front_addr":"172.21.15.122:6805/2404754587","state":["exists","up"]},{"osd":5,"uuid":"a7f1d757-75f9-4972-8f57-98a6452ab1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":708363020},{"type":"v1","addr":"172.21.15.122:6809","nonce":708363020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":708363020},{"type":"v1","addr":"172.21.15.122:6811","nonce":708363020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":708363020},{"type":"v1","addr":"172.21.15.122:6815","nonce":708363020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":708363020},{"type":"v1","addr":"172.21.15.122:6813","nonce":708363020}]},"public_addr":"172.21.15.122:6809/708363020","cluster_addr":"172.21.15.122:6811/708363020","heartbeat_back_addr":"172.21.15.122:6815/708363020","heartbeat_front_addr":"172.21.15.122:6813/708363020","state":["exists","up"]},{"osd":6,"uuid":"22883c68-5a21-46fb-aecc-10e654a8e2ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":2302023781},{"type":"v1","addr":"172.21.15.122:6817","nonce":2302023781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":2302023781},{"type":"v1","addr":"172.21.15.122:6819","nonce":2302023781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":2302023781},{"type":"v1","addr":"172.21.15.122:6823","nonce":2302023781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":2302023781},{"type":"v1","addr":"172.21.15.122:6821","nonce":2302023781}]},"public_addr":"172.21.15.122:6817/2302023781","cluster_addr":"172.21.15.122:6819/2302023781","heartbeat_back_addr":"172.21.15.122:6823/2302023781","heartbeat_front_addr":"172.21.15.122:6821/2302023781","state":["exists","up"]},{"osd":7,"uuid":"db0763d2-09d4-44e9-ae0b-9d6b43d363d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":588529180},{"type":"v1","addr":"172.21.15.122:6825","nonce":588529180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":588529180},{"type":"v1","addr":"172.21.15.122:6827","nonce":588529180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":588529180},{"type":"v1","addr":"172.21.15.122:6831","nonce":588529180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":588529180},{"type":"v1","addr":"172.21.15.122:6829","nonce":588529180}]},"public_addr":"172.21.15.122:6825/588529180","cluster_addr":"172.21.15.122:6827/588529180","heartbeat_back_addr":"172.21.15.122:6831/588529180","heartbeat_front_addr":"172.21.15.122:6829/588529180","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:37:59.442507+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:38:30.071594+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:38:55.953833+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:39:21.900881+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:39:43.074154+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:40:07.139023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:40:30.598928+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:40:55.305914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:0/773205658":"2024-04-03T04:36:51.400435+0000","172.21.15.53:6800/1928933950":"2024-04-03T04:36:00.417623+0000","172.21.15.53:6801/345491441":"2024-04-03T04:36:51.400435+0000","172.21.15.53:0/1377540658":"2024-04-03T04:36:00.417623+0000","172.21.15.53:0/644794640":"2024-04-03T04:36:00.417623+0000","172.21.15.53:0/65784368":"2024-04-03T04:35:31.902665+0000","172.21.15.53:0/225187350":"2024-04-03T04:36:51.400435+0000","172.21.15.53:6801/1928933950":"2024-04-03T04:36:00.417623+0000","172.21.15.53:0/984413801":"2024-04-03T04:35:31.902665+0000","172.21.15.53:6800/345491441":"2024-04-03T04:36:51.400435+0000","172.21.15.53:6800/1387513254":"2024-04-03T04:35:31.902665+0000","172.21.15.53:6801/1387513254":"2024-04-03T04:35:31.902665+0000","172.21.15.53:0/445047527":"2024-04-03T04:36:51.400435+0000","172.21.15.53:0/92223541":"2024-04-03T04:36:00.417623+0000","172.21.15.53:0/1772532444":"2024-04-03T04:35:31.902665+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T04:41:10.786 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:10 smithi053 ceph-mon[24889]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-04-02T04:41:10.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:10 smithi122 ceph-mon[27145]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-04-02T04:41:11.518 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-02T04:41:11.519 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-02T04:41:11.742 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/mon.smithi053/config 2024-04-02T04:41:11.796 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:11 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/3310528697' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T04:41:11.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:11 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/3310528697' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T04:41:12.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:12 smithi053 ceph-mon[24889]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-04-02T04:41:12.815 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:12 smithi122 ceph-mon[27145]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-04-02T04:41:13.486 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-02T04:41:13.487 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":59,"fsid":"48e16742-f0aa-11ee-b647-cb9ed24678a4","created":"2024-04-02T04:34:58.695179+0000","modified":"2024-04-02T04:41:00.940645+0000","last_up_change":"2024-04-02T04:40:56.944274+0000","last_in_change":"2024-04-02T04:40:39.849253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T04:38:57.767014+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"98971ffe-6ad2-4adc-8d2d-7b19f4efc239","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1162124435},{"type":"v1","addr":"172.21.15.53:6803","nonce":1162124435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1162124435},{"type":"v1","addr":"172.21.15.53:6805","nonce":1162124435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":1162124435},{"type":"v1","addr":"172.21.15.53:6809","nonce":1162124435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1162124435},{"type":"v1","addr":"172.21.15.53:6807","nonce":1162124435}]},"public_addr":"172.21.15.53:6803/1162124435","cluster_addr":"172.21.15.53:6805/1162124435","heartbeat_back_addr":"172.21.15.53:6809/1162124435","heartbeat_front_addr":"172.21.15.53:6807/1162124435","state":["exists","up"]},{"osd":1,"uuid":"1ca259ef-434c-4f49-8daa-c00ad50398c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2808409208},{"type":"v1","addr":"172.21.15.53:6811","nonce":2808409208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2808409208},{"type":"v1","addr":"172.21.15.53:6813","nonce":2808409208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":2808409208},{"type":"v1","addr":"172.21.15.53:6817","nonce":2808409208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2808409208},{"type":"v1","addr":"172.21.15.53:6815","nonce":2808409208}]},"public_addr":"172.21.15.53:6811/2808409208","cluster_addr":"172.21.15.53:6813/2808409208","heartbeat_back_addr":"172.21.15.53:6817/2808409208","heartbeat_front_addr":"172.21.15.53:6815/2808409208","state":["exists","up"]},{"osd":2,"uuid":"ab3c0752-d41b-4164-9335-5aced865ae2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":2977813099},{"type":"v1","addr":"172.21.15.53:6819","nonce":2977813099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":2977813099},{"type":"v1","addr":"172.21.15.53:6821","nonce":2977813099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2977813099},{"type":"v1","addr":"172.21.15.53:6825","nonce":2977813099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":2977813099},{"type":"v1","addr":"172.21.15.53:6823","nonce":2977813099}]},"public_addr":"172.21.15.53:6819/2977813099","cluster_addr":"172.21.15.53:6821/2977813099","heartbeat_back_addr":"172.21.15.53:6825/2977813099","heartbeat_front_addr":"172.21.15.53:6823/2977813099","state":["exists","up"]},{"osd":3,"uuid":"1c720f15-3974-4e40-8ce0-deaa392dcce9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3564481581},{"type":"v1","addr":"172.21.15.53:6827","nonce":3564481581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3564481581},{"type":"v1","addr":"172.21.15.53:6829","nonce":3564481581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3564481581},{"type":"v1","addr":"172.21.15.53:6833","nonce":3564481581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3564481581},{"type":"v1","addr":"172.21.15.53:6831","nonce":3564481581}]},"public_addr":"172.21.15.53:6827/3564481581","cluster_addr":"172.21.15.53:6829/3564481581","heartbeat_back_addr":"172.21.15.53:6833/3564481581","heartbeat_front_addr":"172.21.15.53:6831/3564481581","state":["exists","up"]},{"osd":4,"uuid":"57d48c55-b883-4458-824e-8505ee3b7916","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":2404754587},{"type":"v1","addr":"172.21.15.122:6801","nonce":2404754587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":2404754587},{"type":"v1","addr":"172.21.15.122:6803","nonce":2404754587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":2404754587},{"type":"v1","addr":"172.21.15.122:6807","nonce":2404754587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":2404754587},{"type":"v1","addr":"172.21.15.122:6805","nonce":2404754587}]},"public_addr":"172.21.15.122:6801/2404754587","cluster_addr":"172.21.15.122:6803/2404754587","heartbeat_back_addr":"172.21.15.122:6807/2404754587","heartbeat_front_addr":"172.21.15.122:6805/2404754587","state":["exists","up"]},{"osd":5,"uuid":"a7f1d757-75f9-4972-8f57-98a6452ab1a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":708363020},{"type":"v1","addr":"172.21.15.122:6809","nonce":708363020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":708363020},{"type":"v1","addr":"172.21.15.122:6811","nonce":708363020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":708363020},{"type":"v1","addr":"172.21.15.122:6815","nonce":708363020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":708363020},{"type":"v1","addr":"172.21.15.122:6813","nonce":708363020}]},"public_addr":"172.21.15.122:6809/708363020","cluster_addr":"172.21.15.122:6811/708363020","heartbeat_back_addr":"172.21.15.122:6815/708363020","heartbeat_front_addr":"172.21.15.122:6813/708363020","state":["exists","up"]},{"osd":6,"uuid":"22883c68-5a21-46fb-aecc-10e654a8e2ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":2302023781},{"type":"v1","addr":"172.21.15.122:6817","nonce":2302023781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":2302023781},{"type":"v1","addr":"172.21.15.122:6819","nonce":2302023781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":2302023781},{"type":"v1","addr":"172.21.15.122:6823","nonce":2302023781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":2302023781},{"type":"v1","addr":"172.21.15.122:6821","nonce":2302023781}]},"public_addr":"172.21.15.122:6817/2302023781","cluster_addr":"172.21.15.122:6819/2302023781","heartbeat_back_addr":"172.21.15.122:6823/2302023781","heartbeat_front_addr":"172.21.15.122:6821/2302023781","state":["exists","up"]},{"osd":7,"uuid":"db0763d2-09d4-44e9-ae0b-9d6b43d363d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":588529180},{"type":"v1","addr":"172.21.15.122:6825","nonce":588529180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":588529180},{"type":"v1","addr":"172.21.15.122:6827","nonce":588529180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":588529180},{"type":"v1","addr":"172.21.15.122:6831","nonce":588529180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":588529180},{"type":"v1","addr":"172.21.15.122:6829","nonce":588529180}]},"public_addr":"172.21.15.122:6825/588529180","cluster_addr":"172.21.15.122:6827/588529180","heartbeat_back_addr":"172.21.15.122:6831/588529180","heartbeat_front_addr":"172.21.15.122:6829/588529180","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:37:59.442507+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:38:30.071594+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:38:55.953833+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:39:21.900881+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:39:43.074154+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:40:07.139023+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:40:30.598928+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T04:40:55.305914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:0/773205658":"2024-04-03T04:36:51.400435+0000","172.21.15.53:6800/1928933950":"2024-04-03T04:36:00.417623+0000","172.21.15.53:6801/345491441":"2024-04-03T04:36:51.400435+0000","172.21.15.53:0/1377540658":"2024-04-03T04:36:00.417623+0000","172.21.15.53:0/644794640":"2024-04-03T04:36:00.417623+0000","172.21.15.53:0/65784368":"2024-04-03T04:35:31.902665+0000","172.21.15.53:0/225187350":"2024-04-03T04:36:51.400435+0000","172.21.15.53:6801/1928933950":"2024-04-03T04:36:00.417623+0000","172.21.15.53:0/984413801":"2024-04-03T04:35:31.902665+0000","172.21.15.53:6800/345491441":"2024-04-03T04:36:51.400435+0000","172.21.15.53:6800/1387513254":"2024-04-03T04:35:31.902665+0000","172.21.15.53:6801/1387513254":"2024-04-03T04:35:31.902665+0000","172.21.15.53:0/445047527":"2024-04-03T04:36:51.400435+0000","172.21.15.53:0/92223541":"2024-04-03T04:36:00.417623+0000","172.21.15.53:0/1772532444":"2024-04-03T04:35:31.902665+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T04:41:14.178 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:13 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/812179333' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T04:41:14.178 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:13 smithi053 ceph-mon[24889]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-02T04:41:14.204 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph tell osd.0 flush_pg_stats 2024-04-02T04:41:14.204 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph tell osd.1 flush_pg_stats 2024-04-02T04:41:14.205 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph tell osd.2 flush_pg_stats 2024-04-02T04:41:14.205 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph tell osd.3 flush_pg_stats 2024-04-02T04:41:14.205 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph tell osd.4 flush_pg_stats 2024-04-02T04:41:14.206 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph tell osd.5 flush_pg_stats 2024-04-02T04:41:14.206 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph tell osd.6 flush_pg_stats 2024-04-02T04:41:14.207 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph tell osd.7 flush_pg_stats 2024-04-02T04:41:14.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:13 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/812179333' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T04:41:14.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:13 smithi122 ceph-mon[27145]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-02T04:41:14.620 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/mon.smithi053/config 2024-04-02T04:41:14.620 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/mon.smithi053/config 2024-04-02T04:41:14.622 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/mon.smithi053/config 2024-04-02T04:41:14.623 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/mon.smithi053/config 2024-04-02T04:41:14.629 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/mon.smithi053/config 2024-04-02T04:41:14.635 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/mon.smithi053/config 2024-04-02T04:41:14.649 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/mon.smithi053/config 2024-04-02T04:41:14.650 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/mon.smithi053/config 2024-04-02T04:41:15.695 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:15 smithi053 ceph-mon[24889]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:15.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:15 smithi122 ceph-mon[27145]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:17.726 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:17 smithi053 ceph-mon[24889]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:17.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:17 smithi122 ceph-mon[27145]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:19.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:19 smithi122 ceph-mon[27145]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:19.983 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:19 smithi053 ceph-mon[24889]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:21.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:21 smithi053 ceph-mon[24889]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:21.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:21 smithi122 ceph-mon[27145]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:24.244 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:23 smithi053 ceph-mon[24889]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:24.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:23 smithi122 ceph-mon[27145]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:25.455 INFO:teuthology.orchestra.run.smithi053.stdout:167503724561 2024-04-02T04:41:25.455 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.5 2024-04-02T04:41:25.752 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:25 smithi053 ceph-mon[24889]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:25.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:25 smithi122 ceph-mon[27145]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:27.202 INFO:teuthology.orchestra.run.smithi053.stdout:133143986198 2024-04-02T04:41:27.202 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.4 2024-04-02T04:41:27.733 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:27 smithi053 ceph-mon[24889]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:27.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:27 smithi122 ceph-mon[27145]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:28.155 INFO:teuthology.orchestra.run.smithi053.stdout:38654705707 2024-04-02T04:41:28.155 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.0 2024-04-02T04:41:28.166 INFO:teuthology.orchestra.run.smithi053.stdout:201863462925 2024-04-02T04:41:28.166 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.6 2024-04-02T04:41:28.807 INFO:teuthology.orchestra.run.smithi053.stdout:111669149723 2024-04-02T04:41:28.807 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.3 2024-04-02T04:41:29.454 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/mon.smithi053/config 2024-04-02T04:41:29.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:29 smithi053 ceph-mon[24889]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:29.833 INFO:teuthology.orchestra.run.smithi053.stdout:236223201289 2024-04-02T04:41:29.833 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.7 2024-04-02T04:41:29.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:29 smithi122 ceph-mon[27145]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:30.646 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/mon.smithi053/config 2024-04-02T04:41:30.646 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/mon.smithi053/config 2024-04-02T04:41:30.646 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/mon.smithi053/config 2024-04-02T04:41:31.022 INFO:teuthology.orchestra.run.smithi053.stdout:81604378658 2024-04-02T04:41:31.022 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.2 2024-04-02T04:41:31.182 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/mon.smithi053/config 2024-04-02T04:41:31.610 INFO:teuthology.orchestra.run.smithi053.stdout:60129542183 2024-04-02T04:41:31.610 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.1 2024-04-02T04:41:31.623 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/mon.smithi053/config 2024-04-02T04:41:31.806 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:31 smithi053 ceph-mon[24889]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:31.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:31 smithi122 ceph-mon[27145]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:34.245 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:33 smithi053 ceph-mon[24889]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:34.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:33 smithi122 ceph-mon[27145]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:35.585 INFO:teuthology.orchestra.run.smithi053.stdout:133143986200 2024-04-02T04:41:35.806 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:35 smithi053 ceph-mon[24889]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:35.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:35 smithi122 ceph-mon[27145]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:36.806 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:36 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/1317925538' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-02T04:41:36.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:36 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/1317925538' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-02T04:41:37.013 INFO:teuthology.orchestra.run.smithi053.stdout:38654705710 2024-04-02T04:41:37.630 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705707 got 38654705710 for osd.0 2024-04-02T04:41:37.631 DEBUG:teuthology.parallel:result is None 2024-04-02T04:41:37.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:37 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/502722957' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-02T04:41:37.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:37 smithi053 ceph-mon[24889]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:37.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:37 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/502722957' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-02T04:41:37.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:37 smithi122 ceph-mon[27145]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:39.201 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/mon.smithi053/config 2024-04-02T04:41:39.201 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/mon.smithi053/config 2024-04-02T04:41:39.858 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:39 smithi053 ceph-mon[24889]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:39.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:39 smithi122 ceph-mon[27145]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:41.225 INFO:teuthology.orchestra.run.smithi053.stdout:111669149726 2024-04-02T04:41:41.556 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:41 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/2817315024' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-02T04:41:41.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:41 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/2817315024' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-02T04:41:41.928 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149723 got 111669149726 for osd.3 2024-04-02T04:41:41.928 DEBUG:teuthology.parallel:result is None 2024-04-02T04:41:42.300 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986198 got 133143986200 for osd.4 2024-04-02T04:41:42.300 DEBUG:teuthology.parallel:result is None 2024-04-02T04:41:42.543 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:42 smithi053 ceph-mon[24889]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:42.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:42 smithi122 ceph-mon[27145]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:42.841 INFO:teuthology.orchestra.run.smithi053.stdout:167503724565 2024-04-02T04:41:43.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:43 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/1659473520' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-02T04:41:43.556 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:43 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/1659473520' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-02T04:41:43.631 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724561 got 167503724565 for osd.5 2024-04-02T04:41:43.631 DEBUG:teuthology.parallel:result is None 2024-04-02T04:41:44.133 INFO:teuthology.orchestra.run.smithi053.stdout:236223201291 2024-04-02T04:41:44.332 INFO:teuthology.orchestra.run.smithi053.stdout:201863462929 2024-04-02T04:41:44.556 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:44 smithi053 ceph-mon[24889]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:44.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:44 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/1754104317' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-02T04:41:44.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:44 smithi122 ceph-mon[27145]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:44.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:44 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/1754104317' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-02T04:41:44.954 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462925 got 201863462929 for osd.6 2024-04-02T04:41:44.955 DEBUG:teuthology.parallel:result is None 2024-04-02T04:41:45.518 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:45 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/3421694515' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-02T04:41:45.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:45 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/3421694515' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-02T04:41:46.162 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201289 got 236223201291 for osd.7 2024-04-02T04:41:46.162 DEBUG:teuthology.parallel:result is None 2024-04-02T04:41:46.388 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:46 smithi053 ceph-mon[24889]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:46.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:46 smithi122 ceph-mon[27145]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:46.987 INFO:teuthology.orchestra.run.smithi053.stdout:60129542186 2024-04-02T04:41:47.061 INFO:teuthology.orchestra.run.smithi053.stdout:81604378661 2024-04-02T04:41:47.488 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:47 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/234175295' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-02T04:41:47.489 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:47 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/1038144564' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-02T04:41:47.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:47 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/234175295' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-02T04:41:47.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:47 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/1038144564' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-02T04:41:47.808 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378658 got 81604378661 for osd.2 2024-04-02T04:41:47.808 DEBUG:teuthology.parallel:result is None 2024-04-02T04:41:48.219 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542183 got 60129542186 for osd.1 2024-04-02T04:41:48.219 DEBUG:teuthology.parallel:result is None 2024-04-02T04:41:48.219 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-02T04:41:48.219 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-02T04:41:48.447 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/mon.smithi053/config 2024-04-02T04:41:48.550 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:48 smithi053 ceph-mon[24889]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:48.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:48 smithi122 ceph-mon[27145]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:49.831 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:49 smithi053 ceph-mon[24889]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:49.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:49 smithi122 ceph-mon[27145]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:50.226 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-02T04:41:50.228 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-04-02T04:41:50.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:50 smithi053 ceph-mon[24889]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:41:50.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:50 smithi122 ceph-mon[27145]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:41:50.928 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-04-02T04:41:49.503782+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48736,"kb_used_data":6304,"kb_used_omap":0,"kb_used_meta":42368,"kb_avail":749912480,"statfs":{"total":767960285184,"available":767910379520,"internally_reserved":0,"allocated":6455296,"data_stored":3546942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43384832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002250"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":226,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-02T04:41:00.946314+0000","last_change":"2024-04-02T04:41:00.946314+0000","last_active":"2024-04-02T04:41:00.946314+0000","last_peered":"2024-04-02T04:41:00.946314+0000","last_clean":"2024-04-02T04:41:00.946314+0000","last_became_active":"2024-04-02T04:41:00.945895+0000","last_became_peered":"2024-04-02T04:41:00.945895+0000","last_unstale":"2024-04-02T04:41:00.946314+0000","last_undegraded":"2024-04-02T04:41:00.946314+0000","last_fullsized":"2024-04-02T04:41:00.946314+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:38:58.021667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:58.021667+0000","last_clean_scrub_stamp":"2024-04-02T04:38:58.021667+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:05:48.012357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201293,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6048,"kb_used_data":1112,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739104,"statfs":{"total":95995035648,"available":95988842496,"internally_reserved":0,"allocated":1138688,"data_stored":773838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]}]},{"osd":6,"up_from":47,"seq":201863462930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6108,"kb_used_data":1108,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739044,"statfs":{"total":95995035648,"available":95988781056,"internally_reserved":0,"allocated":1134592,"data_stored":773573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 04:41:33 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":7.5759999999999996,"5min":7.5759999999999996,"15min":7.5759999999999996},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":7.5640000000000001,"5min":7.5640000000000001,"15min":7.5640000000000001},"last":0.53300000000000003}]},{"osd":1,"last update":"Tue Apr 2 04:41:33 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.437},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.502}]},{"osd":2,"last update":"Tue Apr 2 04:41:33 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.495},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.51800000000000002}]},{"osd":3,"last update":"Tue Apr 2 04:41:33 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.56000000000000005}]},{"osd":4,"last update":"Tue Apr 2 04:41:33 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.41699999999999998}]},{"osd":5,"last update":"Tue Apr 2 04:41:33 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.54400000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542187,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6360,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738792,"statfs":{"total":95995035648,"available":95988523008,"internally_reserved":0,"allocated":606208,"data_stored":244968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 04:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.37,"15min":0.34599999999999997},"min":{"1min":0.251,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.35599999999999998,"15min":0.32700000000000001},"min":{"1min":0.29599999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.53000000000000003,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.40999999999999998}]},{"osd":2,"last update":"Tue Apr 2 04:41:00 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.40200000000000002,"15min":0.38700000000000001},"min":{"1min":0.32600000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"last":0.377},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.39900000000000002,"15min":0.39100000000000001},"min":{"1min":0.32900000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.42699999999999999}]},{"osd":3,"last update":"Tue Apr 2 04:41:29 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.496,"15min":0.502},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.89700000000000002,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.504,"5min":0.51800000000000002,"15min":0.52000000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.51700000000000002}]},{"osd":4,"last update":"Tue Apr 2 04:40:46 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.68899999999999995}]},{"osd":5,"last update":"Tue Apr 2 04:41:09 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.59399999999999997}]},{"osd":6,"last update":"Tue Apr 2 04:41:33 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.629}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705713,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6884,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738268,"statfs":{"total":95995035648,"available":95987986432,"internally_reserved":0,"allocated":1142784,"data_stored":774161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 04:41:35 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35099999999999998,"15min":0.33300000000000002},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.53800000000000003,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.34200000000000003,"15min":0.32600000000000001},"min":{"1min":0.24399999999999999,"5min":0.224,"15min":0.224},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.42499999999999999}]},{"osd":2,"last update":"Tue Apr 2 04:41:05 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.376,"15min":0.36699999999999999},"min":{"1min":0.27700000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.39600000000000002,"15min":0.38600000000000001},"min":{"1min":0.29499999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.44700000000000001}]},{"osd":3,"last update":"Tue Apr 2 04:41:27 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.45000000000000001,"15min":0.45200000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.48499999999999999,"15min":0.48699999999999999},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.47899999999999998}]},{"osd":4,"last update":"Tue Apr 2 04:40:47 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.77000000000000002}]},{"osd":5,"last update":"Tue Apr 2 04:41:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.78700000000000003}]},{"osd":6,"last update":"Tue Apr 2 04:41:35 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.69999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81599999999999995}]}]},{"osd":2,"up_from":19,"seq":81604378662,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6236,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738916,"statfs":{"total":95995035648,"available":95988649984,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 04:41:01 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.35099999999999998,"15min":0.33900000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.438,"5min":0.38,"15min":0.37},"min":{"1min":0.29599999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.41399999999999998}]},{"osd":1,"last update":"Tue Apr 2 04:41:01 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.36599999999999999,"15min":0.36199999999999999},"min":{"1min":0.28199999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.39100000000000001,"15min":0.38},"min":{"1min":0.32100000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.34899999999999998}]},{"osd":3,"last update":"Tue Apr 2 04:41:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.42299999999999999,"15min":0.42399999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.54100000000000004,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.372}]},{"osd":4,"last update":"Tue Apr 2 04:40:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.61199999999999999}]},{"osd":5,"last update":"Tue Apr 2 04:41:10 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.65800000000000003}]},{"osd":6,"last update":"Tue Apr 2 04:41:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":10.65,"5min":10.65,"15min":10.65},"last":0.621},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.393,"5min":1.393,"15min":1.393},"last":0.55600000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]}]},{"osd":3,"up_from":26,"seq":111669149728,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5720,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739432,"statfs":{"total":95995035648,"available":95989178368,"internally_reserved":0,"allocated":606208,"data_stored":244968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 04:41:29 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.38200000000000001,"15min":0.38},"min":{"1min":0.26800000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.438},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.40799999999999997,"15min":0.40400000000000003},"min":{"1min":0.31900000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.66700000000000004,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.33500000000000002}]},{"osd":1,"last update":"Tue Apr 2 04:41:29 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.42599999999999999,"15min":0.42399999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.59799999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.432,"5min":0.40999999999999998,"15min":0.40600000000000003},"min":{"1min":0.308,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.34999999999999998}]},{"osd":2,"last update":"Tue Apr 2 04:41:29 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.48699999999999999,"15min":0.48299999999999998},"min":{"1min":0.376,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.46000000000000002,"15min":0.45800000000000002},"min":{"1min":0.35399999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.39000000000000001}]},{"osd":4,"last update":"Tue Apr 2 04:41:47 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.56000000000000005,"15min":0.55800000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.56200000000000006,"15min":0.55800000000000005},"min":{"1min":0.46100000000000002,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.746,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.61499999999999999}]},{"osd":5,"last update":"Tue Apr 2 04:41:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.73799999999999999}]},{"osd":6,"last update":"Tue Apr 2 04:41:35 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.67900000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]}]},{"osd":4,"up_from":31,"seq":133143986202,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5724,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739428,"statfs":{"total":95995035648,"available":95989174272,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 04:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.52000000000000002}]},{"osd":1,"last update":"Tue Apr 2 04:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.40100000000000002}]},{"osd":2,"last update":"Tue Apr 2 04:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.42299999999999999}]},{"osd":3,"last update":"Tue Apr 2 04:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.44800000000000001}]},{"osd":5,"last update":"Tue Apr 2 04:41:08 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.505},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.41199999999999998}]},{"osd":6,"last update":"Tue Apr 2 04:41:33 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.37},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.45700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]}]},{"osd":5,"up_from":39,"seq":167503724567,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5656,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739496,"statfs":{"total":95995035648,"available":95989243904,"internally_reserved":0,"allocated":606208,"data_stored":244968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 04:41:08 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.57999999999999996}]},{"osd":1,"last update":"Tue Apr 2 04:41:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.629}]},{"osd":2,"last update":"Tue Apr 2 04:41:08 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.61899999999999999}]},{"osd":3,"last update":"Tue Apr 2 04:41:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.68400000000000005}]},{"osd":4,"last update":"Tue Apr 2 04:41:08 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.437,"15min":0.437},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.51200000000000001}]},{"osd":6,"last update":"Tue Apr 2 04:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.59599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T04:41:50.930 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-02T04:41:51.150 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/mon.smithi053/config 2024-04-02T04:41:51.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:51 smithi122 ceph-mon[27145]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:51.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:51 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/trash_purge_schedule"}]: dispatch 2024-04-02T04:41:51.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:51 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:41:51.944 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:51 smithi053 ceph-mon[24889]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:51.945 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:51 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/trash_purge_schedule"}]: dispatch 2024-04-02T04:41:51.945 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:51 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:41:52.848 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-02T04:41:52.849 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-04-02T04:41:53.449 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-04-02T04:41:51.504089+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48736,"kb_used_data":6304,"kb_used_omap":0,"kb_used_meta":42368,"kb_avail":749912480,"statfs":{"total":767960285184,"available":767910379520,"internally_reserved":0,"allocated":6455296,"data_stored":3546942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43384832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002229"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":226,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-02T04:41:00.946314+0000","last_change":"2024-04-02T04:41:00.946314+0000","last_active":"2024-04-02T04:41:00.946314+0000","last_peered":"2024-04-02T04:41:00.946314+0000","last_clean":"2024-04-02T04:41:00.946314+0000","last_became_active":"2024-04-02T04:41:00.945895+0000","last_became_peered":"2024-04-02T04:41:00.945895+0000","last_unstale":"2024-04-02T04:41:00.946314+0000","last_undegraded":"2024-04-02T04:41:00.946314+0000","last_fullsized":"2024-04-02T04:41:00.946314+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T04:38:58.021667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T04:38:58.021667+0000","last_clean_scrub_stamp":"2024-04-02T04:38:58.021667+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T08:05:48.012357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201293,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6048,"kb_used_data":1112,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739104,"statfs":{"total":95995035648,"available":95988842496,"internally_reserved":0,"allocated":1138688,"data_stored":773838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]}]},{"osd":6,"up_from":47,"seq":201863462930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6108,"kb_used_data":1108,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739044,"statfs":{"total":95995035648,"available":95988781056,"internally_reserved":0,"allocated":1134592,"data_stored":773573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 04:41:33 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":7.5759999999999996,"5min":7.5759999999999996,"15min":7.5759999999999996},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":7.5640000000000001,"5min":7.5640000000000001,"15min":7.5640000000000001},"last":0.53300000000000003}]},{"osd":1,"last update":"Tue Apr 2 04:41:33 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.437},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.502}]},{"osd":2,"last update":"Tue Apr 2 04:41:33 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.495},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.51800000000000002}]},{"osd":3,"last update":"Tue Apr 2 04:41:33 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.56000000000000005}]},{"osd":4,"last update":"Tue Apr 2 04:41:33 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.41699999999999998}]},{"osd":5,"last update":"Tue Apr 2 04:41:33 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.54400000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542187,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6360,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738792,"statfs":{"total":95995035648,"available":95988523008,"internally_reserved":0,"allocated":606208,"data_stored":244968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 04:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.37,"15min":0.34599999999999997},"min":{"1min":0.251,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.35599999999999998,"15min":0.32700000000000001},"min":{"1min":0.29599999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.53000000000000003,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.40999999999999998}]},{"osd":2,"last update":"Tue Apr 2 04:41:00 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.40200000000000002,"15min":0.38700000000000001},"min":{"1min":0.32600000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"last":0.377},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.39900000000000002,"15min":0.39100000000000001},"min":{"1min":0.32900000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.42699999999999999}]},{"osd":3,"last update":"Tue Apr 2 04:41:29 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.496,"15min":0.502},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.89700000000000002,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.504,"5min":0.51800000000000002,"15min":0.52000000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.51700000000000002}]},{"osd":4,"last update":"Tue Apr 2 04:40:46 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.68899999999999995}]},{"osd":5,"last update":"Tue Apr 2 04:41:09 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.59399999999999997}]},{"osd":6,"last update":"Tue Apr 2 04:41:33 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.629}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705713,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6884,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738268,"statfs":{"total":95995035648,"available":95987986432,"internally_reserved":0,"allocated":1142784,"data_stored":774161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 2 04:41:35 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35099999999999998,"15min":0.33300000000000002},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.53800000000000003,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.34200000000000003,"15min":0.32600000000000001},"min":{"1min":0.24399999999999999,"5min":0.224,"15min":0.224},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.42499999999999999}]},{"osd":2,"last update":"Tue Apr 2 04:41:05 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.376,"15min":0.36699999999999999},"min":{"1min":0.27700000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.39600000000000002,"15min":0.38600000000000001},"min":{"1min":0.29499999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.44700000000000001}]},{"osd":3,"last update":"Tue Apr 2 04:41:27 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.45000000000000001,"15min":0.45200000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.48499999999999999,"15min":0.48699999999999999},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.47899999999999998}]},{"osd":4,"last update":"Tue Apr 2 04:40:47 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.77000000000000002}]},{"osd":5,"last update":"Tue Apr 2 04:41:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.78700000000000003}]},{"osd":6,"last update":"Tue Apr 2 04:41:35 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.69999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81599999999999995}]}]},{"osd":2,"up_from":19,"seq":81604378662,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6236,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738916,"statfs":{"total":95995035648,"available":95988649984,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 04:41:01 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.35099999999999998,"15min":0.33900000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.438,"5min":0.38,"15min":0.37},"min":{"1min":0.29599999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.41399999999999998}]},{"osd":1,"last update":"Tue Apr 2 04:41:01 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.36599999999999999,"15min":0.36199999999999999},"min":{"1min":0.28199999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.39100000000000001,"15min":0.38},"min":{"1min":0.32100000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.34899999999999998}]},{"osd":3,"last update":"Tue Apr 2 04:41:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.42299999999999999,"15min":0.42399999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.54100000000000004,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.372}]},{"osd":4,"last update":"Tue Apr 2 04:40:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.61199999999999999}]},{"osd":5,"last update":"Tue Apr 2 04:41:10 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.65800000000000003}]},{"osd":6,"last update":"Tue Apr 2 04:41:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":10.65,"5min":10.65,"15min":10.65},"last":0.621},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.393,"5min":1.393,"15min":1.393},"last":0.55600000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]}]},{"osd":3,"up_from":26,"seq":111669149728,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5720,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739432,"statfs":{"total":95995035648,"available":95989178368,"internally_reserved":0,"allocated":606208,"data_stored":244968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 04:41:29 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.38200000000000001,"15min":0.38},"min":{"1min":0.26800000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.438},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.40799999999999997,"15min":0.40400000000000003},"min":{"1min":0.31900000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.66700000000000004,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.33500000000000002}]},{"osd":1,"last update":"Tue Apr 2 04:41:29 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.42599999999999999,"15min":0.42399999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.59799999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.432,"5min":0.40999999999999998,"15min":0.40600000000000003},"min":{"1min":0.308,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.34999999999999998}]},{"osd":2,"last update":"Tue Apr 2 04:41:29 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.48699999999999999,"15min":0.48299999999999998},"min":{"1min":0.376,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.46000000000000002,"15min":0.45800000000000002},"min":{"1min":0.35399999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.39000000000000001}]},{"osd":4,"last update":"Tue Apr 2 04:41:47 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.56000000000000005,"15min":0.55800000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.56200000000000006,"15min":0.55800000000000005},"min":{"1min":0.46100000000000002,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.746,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.61499999999999999}]},{"osd":5,"last update":"Tue Apr 2 04:41:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.73799999999999999}]},{"osd":6,"last update":"Tue Apr 2 04:41:35 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.67900000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]}]},{"osd":4,"up_from":31,"seq":133143986203,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5724,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739428,"statfs":{"total":95995035648,"available":95989174272,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 04:41:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56799999999999995,"15min":0.57199999999999995},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.753,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.58099999999999996,"15min":0.57899999999999996},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.80000000000000004,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.73399999999999999}]},{"osd":1,"last update":"Tue Apr 2 04:41:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.52700000000000002,"15min":0.52500000000000002},"min":{"1min":0.38600000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.72599999999999998,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.35899999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.81100000000000005,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.63800000000000001}]},{"osd":2,"last update":"Tue Apr 2 04:41:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53400000000000003,"15min":0.53300000000000003},"min":{"1min":0.34999999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.621},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.55300000000000005,"15min":0.55400000000000005},"min":{"1min":0.36199999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74099999999999999,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.70399999999999996}]},{"osd":3,"last update":"Tue Apr 2 04:41:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54900000000000004,"15min":0.54800000000000004},"min":{"1min":0.441,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.73799999999999999,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.60499999999999998}]},{"osd":5,"last update":"Tue Apr 2 04:41:08 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.499},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.69099999999999995}]},{"osd":6,"last update":"Tue Apr 2 04:41:33 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.63}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]}]},{"osd":5,"up_from":39,"seq":167503724567,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5656,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739496,"statfs":{"total":95995035648,"available":95989243904,"internally_reserved":0,"allocated":606208,"data_stored":244968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 2 04:41:08 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.57999999999999996}]},{"osd":1,"last update":"Tue Apr 2 04:41:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.629}]},{"osd":2,"last update":"Tue Apr 2 04:41:08 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.61899999999999999}]},{"osd":3,"last update":"Tue Apr 2 04:41:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.68400000000000005}]},{"osd":4,"last update":"Tue Apr 2 04:41:08 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.437,"15min":0.437},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.51200000000000001}]},{"osd":6,"last update":"Tue Apr 2 04:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.59599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T04:41:53.451 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-02T04:41:53.451 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-02T04:41:53.451 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-02T04:41:53.451 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph health --format=json 2024-04-02T04:41:53.672 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/mon.smithi053/config 2024-04-02T04:41:54.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:53 smithi053 ceph-mon[24889]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:41:54.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:53 smithi053 ceph-mon[24889]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:54.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:53 smithi122 ceph-mon[27145]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:41:54.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:53 smithi122 ceph-mon[27145]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:55.462 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-02T04:41:55.462 INFO:teuthology.orchestra.run.smithi053.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-02T04:41:55.763 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:55 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/945032475' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-02T04:41:55.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:55 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/945032475' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-02T04:41:56.071 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-02T04:41:56.071 INFO:tasks.cephadm:Setup complete, yielding 2024-04-02T04:41:56.071 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-02T04:41:56.086 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi053.front.sepia.ceph.com 2024-04-02T04:41:56.087 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch status' 2024-04-02T04:41:56.734 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:56 smithi053 ceph-mon[24889]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:56.897 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:56 smithi122 ceph-mon[27145]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:58.145 INFO:teuthology.orchestra.run.smithi053.stdout:Backend: cephadm 2024-04-02T04:41:58.145 INFO:teuthology.orchestra.run.smithi053.stdout:Available: Yes 2024-04-02T04:41:58.145 INFO:teuthology.orchestra.run.smithi053.stdout:Paused: No 2024-04-02T04:41:58.718 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:58 smithi053 ceph-mon[24889]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:58.741 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch ps' 2024-04-02T04:41:58.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:58 smithi122 ceph-mon[27145]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:41:59.716 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:59 smithi053 ceph-mon[24889]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:41:59.716 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:59 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:41:59.716 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:59 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:41:59.716 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:59 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:41:59.716 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:41:59 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:41:59.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:59 smithi122 ceph-mon[27145]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:41:59.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:59 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:41:59.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:59 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:41:59.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:59 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:41:59.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:41:59 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:00.699 INFO:teuthology.orchestra.run.smithi053.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-02T04:42:00.699 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager.smithi053 smithi053 *:9093,9094 running (4m) 2m ago 5m 15.5M - ba2b418f427c 8d27e10ebf84 2024-04-02T04:42:00.699 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi053 smithi053 running (5m) 2m ago 5m 6409k - 17.2.0 e1d6a67b021e a5c436bbd36d 2024-04-02T04:42:00.699 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi122 smithi122 running (4m) 65s ago 4m 6405k - 17.2.0 e1d6a67b021e b7074474cd46 2024-04-02T04:42:00.699 INFO:teuthology.orchestra.run.smithi053.stdout:grafana.smithi053 smithi053 *:3000 running (4m) 2m ago 5m 42.7M - 8.3.5 dad864ee21e9 8f4a8d871d62 2024-04-02T04:42:00.699 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi053.legdmv smithi053 *:9283 running (6m) 2m ago 6m 447M - 17.2.0 e1d6a67b021e 5616241666e4 2024-04-02T04:42:00.699 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi122.cpwlyk smithi122 *:8443,9283 running (4m) 65s ago 4m 412M - 17.2.0 e1d6a67b021e b28b3080c55d 2024-04-02T04:42:00.699 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi053 smithi053 running (6m) 2m ago 7m 42.2M 2048M 17.2.0 e1d6a67b021e 665e4412f6da 2024-04-02T04:42:00.699 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi122 smithi122 running (4m) 65s ago 4m 38.8M 2048M 17.2.0 e1d6a67b021e a2df07e0883d 2024-04-02T04:42:00.699 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi053 smithi053 *:9100 running (5m) 2m ago 5m 19.0M - 1dbe0e931976 87185e572b31 2024-04-02T04:42:00.700 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi122 smithi122 *:9100 running (4m) 65s ago 4m 22.1M - 1dbe0e931976 d576cf080067 2024-04-02T04:42:00.700 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 smithi053 running (4m) 2m ago 4m 32.7M 3217M 17.2.0 e1d6a67b021e 906cb1fd8042 2024-04-02T04:42:00.700 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 smithi053 running (3m) 2m ago 3m 37.2M 3217M 17.2.0 e1d6a67b021e ac1ad6266e63 2024-04-02T04:42:00.700 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 smithi053 running (3m) 2m ago 3m 30.3M 3217M 17.2.0 e1d6a67b021e 86964c8a94ff 2024-04-02T04:42:00.700 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 smithi053 running (2m) 2m ago 2m 12.6M 3217M 17.2.0 e1d6a67b021e 7e8b9e98ae2b 2024-04-02T04:42:00.700 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 smithi122 running (2m) 65s ago 2m 34.0M 3984M 17.2.0 e1d6a67b021e 46076afcd7bc 2024-04-02T04:42:00.700 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 smithi122 running (118s) 65s ago 117s 35.5M 3984M 17.2.0 e1d6a67b021e e687a2f6da4d 2024-04-02T04:42:00.700 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 smithi122 running (94s) 65s ago 94s 30.2M 3984M 17.2.0 e1d6a67b021e 9e371df9a839 2024-04-02T04:42:00.700 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 smithi122 running (69s) 65s ago 69s 10.0M 3984M 17.2.0 e1d6a67b021e 9b3cc398254d 2024-04-02T04:42:00.700 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus.smithi053 smithi053 *:9095 running (4m) 2m ago 4m 53.5M - 514e6a882f6e 36d493ce400e 2024-04-02T04:42:00.806 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:00 smithi053 ceph-mon[24889]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:42:00.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:00 smithi122 ceph-mon[27145]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:42:01.239 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch ls' 2024-04-02T04:42:02.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:02 smithi053 ceph-mon[24889]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:02.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:02 smithi053 ceph-mon[24889]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:42:02.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:02 smithi122 ceph-mon[27145]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:02.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:02 smithi122 ceph-mon[27145]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:42:03.206 INFO:teuthology.orchestra.run.smithi053.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-02T04:42:03.206 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-04-02T04:42:03.206 INFO:teuthology.orchestra.run.smithi053.stdout:crash 2/2 2m ago 6m * 2024-04-02T04:42:03.206 INFO:teuthology.orchestra.run.smithi053.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-04-02T04:42:03.206 INFO:teuthology.orchestra.run.smithi053.stdout:mgr 2/2 2m ago 6m count:2 2024-04-02T04:42:03.206 INFO:teuthology.orchestra.run.smithi053.stdout:mon 2/2 2m ago 5m smithi053:172.21.15.53=smithi053;smithi122:172.21.15.122=smithi122;count:2 2024-04-02T04:42:03.206 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-04-02T04:42:03.207 INFO:teuthology.orchestra.run.smithi053.stdout:osd 8 2m ago - 2024-04-02T04:42:03.207 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-04-02T04:42:03.838 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch host ls' 2024-04-02T04:42:04.056 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:03 smithi053 ceph-mon[24889]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:04.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:03 smithi053 ceph-mon[24889]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:42:04.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:03 smithi122 ceph-mon[27145]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:04.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:03 smithi122 ceph-mon[27145]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:42:05.750 INFO:teuthology.orchestra.run.smithi053.stdout:HOST ADDR LABELS STATUS 2024-04-02T04:42:05.750 INFO:teuthology.orchestra.run.smithi053.stdout:smithi053 172.21.15.53 2024-04-02T04:42:05.750 INFO:teuthology.orchestra.run.smithi053.stdout:smithi122 172.21.15.122 2024-04-02T04:42:05.750 INFO:teuthology.orchestra.run.smithi053.stdout:2 hosts in cluster 2024-04-02T04:42:05.750 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:05 smithi053 ceph-mon[24889]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:42:05.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:05 smithi122 ceph-mon[27145]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:42:06.321 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch device ls' 2024-04-02T04:42:06.843 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:06 smithi053 ceph-mon[24889]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:06.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:06 smithi122 ceph-mon[27145]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:07.832 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:07 smithi053 ceph-mon[24889]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:42:07.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:07 smithi122 ceph-mon[27145]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:42:08.231 INFO:teuthology.orchestra.run.smithi053.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REJECT REASONS 2024-04-02T04:42:08.231 INFO:teuthology.orchestra.run.smithi053.stdout:smithi053 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB751300G8480DGN 480G LVM detected, locked 2024-04-02T04:42:08.231 INFO:teuthology.orchestra.run.smithi053.stdout:smithi122 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204010U400BGN 400G LVM detected, locked 2024-04-02T04:42:08.985 INFO:teuthology.run_tasks:Running task vip.exec... 2024-04-02T04:42:08.994 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi053.front.sepia.ceph.com 2024-04-02T04:42:08.997 DEBUG:teuthology.orchestra.run.smithi053:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-04-02T04:42:09.024 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:08 smithi053 ceph-mon[24889]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:09.027 INFO:teuthology.orchestra.run.smithi053.stderr:+ systemctl stop nfs-server 2024-04-02T04:42:09.034 INFO:tasks.vip:Running commands on role host.b host ubuntu@smithi122.front.sepia.ceph.com 2024-04-02T04:42:09.034 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-04-02T04:42:09.066 INFO:teuthology.orchestra.run.smithi122.stderr:+ systemctl stop nfs-server 2024-04-02T04:42:09.074 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-02T04:42:09.090 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi053.front.sepia.ceph.com 2024-04-02T04:42:09.091 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- bash -c 'ceph fs volume create foofs' 2024-04-02T04:42:09.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:08 smithi122 ceph-mon[27145]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:10.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:09 smithi053 ceph-mon[24889]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:42:10.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:09 smithi122 ceph-mon[27145]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:42:12.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:11 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-04-02T04:42:12.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:11 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-04-02T04:42:12.765 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:12 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi053[24885]: 2024-04-02T04:42:12.429+0000 7fecc2639700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T04:42:12.765 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:12 smithi053 ceph-mon[24889]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:12.765 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:12 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-04-02T04:42:12.765 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:12 smithi053 ceph-mon[24889]: osdmap e60: 8 total, 8 up, 8 in 2024-04-02T04:42:12.765 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:12 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-04-02T04:42:12.765 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:12 smithi053 ceph-mon[24889]: pgmap v208: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:42:12.765 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:12 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-04-02T04:42:12.765 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:12 smithi053 ceph-mon[24889]: osdmap e61: 8 total, 8 up, 8 in 2024-04-02T04:42:12.765 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:12 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-04-02T04:42:12.765 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:12 smithi053 ceph-mon[24889]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T04:42:12.766 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:12 smithi053 ceph-mon[24889]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-02T04:42:12.766 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:12 smithi053 ceph-mon[24889]: osdmap e62: 8 total, 8 up, 8 in 2024-04-02T04:42:12.766 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:12 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-04-02T04:42:12.766 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:12 smithi053 ceph-mon[24889]: fsmap foofs:0 2024-04-02T04:42:12.766 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:12 smithi053 ceph-mon[24889]: Saving service mds.foofs spec with placement count:2 2024-04-02T04:42:12.766 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:12 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:12.766 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:12 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:42:12.766 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:12 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:12.766 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:12 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:42:12.766 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:12 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:12.766 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:12 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:12.766 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:12 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi053.nfrfot", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T04:42:12.766 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:12 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi053.nfrfot", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T04:42:12.766 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:12 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:12.767 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:12 smithi053 ceph-mon[24889]: Deploying daemon mds.foofs.smithi053.nfrfot on smithi053 2024-04-02T04:42:13.104 INFO:teuthology.run_tasks:Running task cephadm.wait_for_service... 2024-04-02T04:42:13.113 INFO:tasks.cephadm:Waiting for ceph service mds.foofs to start (timeout 300)... 2024-04-02T04:42:13.113 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph orch ls -f json 2024-04-02T04:42:13.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:12 smithi122 ceph-mon[27145]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:13.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:12 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-04-02T04:42:13.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:12 smithi122 ceph-mon[27145]: osdmap e60: 8 total, 8 up, 8 in 2024-04-02T04:42:13.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:12 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-04-02T04:42:13.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:12 smithi122 ceph-mon[27145]: pgmap v208: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:42:13.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:12 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-04-02T04:42:13.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:12 smithi122 ceph-mon[27145]: osdmap e61: 8 total, 8 up, 8 in 2024-04-02T04:42:13.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:12 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-04-02T04:42:13.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:12 smithi122 ceph-mon[27145]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T04:42:13.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:12 smithi122 ceph-mon[27145]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-02T04:42:13.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:12 smithi122 ceph-mon[27145]: osdmap e62: 8 total, 8 up, 8 in 2024-04-02T04:42:13.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:12 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-04-02T04:42:13.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:12 smithi122 ceph-mon[27145]: fsmap foofs:0 2024-04-02T04:42:13.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:12 smithi122 ceph-mon[27145]: Saving service mds.foofs spec with placement count:2 2024-04-02T04:42:13.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:12 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:13.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:12 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:42:13.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:12 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:13.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:12 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:42:13.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:12 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:13.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:12 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:13.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:12 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi053.nfrfot", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T04:42:13.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:12 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi053.nfrfot", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T04:42:13.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:12 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:13.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:12 smithi122 ceph-mon[27145]: Deploying daemon mds.foofs.smithi053.nfrfot on smithi053 2024-04-02T04:42:14.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:14 smithi053 ceph-mon[24889]: osdmap e63: 8 total, 8 up, 8 in 2024-04-02T04:42:14.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:14 smithi053 ceph-mon[24889]: pgmap v212: 65 pgs: 20 creating+peering, 33 unknown, 12 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:42:14.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:14 smithi122 ceph-mon[27145]: osdmap e63: 8 total, 8 up, 8 in 2024-04-02T04:42:14.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:14 smithi122 ceph-mon[27145]: pgmap v212: 65 pgs: 20 creating+peering, 33 unknown, 12 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:42:15.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:15 smithi053 ceph-mon[24889]: osdmap e64: 8 total, 8 up, 8 in 2024-04-02T04:42:15.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:15 smithi122 ceph-mon[27145]: osdmap e64: 8 total, 8 up, 8 in 2024-04-02T04:42:16.127 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-02T04:42:16.127 INFO:teuthology.orchestra.run.smithi053.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-02T04:35:49.645669Z", "last_refresh": "2024-04-02T04:39:24.604155Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-02T04:35:44.306903Z", "last_refresh": "2024-04-02T04:39:24.604223Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-02T04:35:47.354203Z", "last_refresh": "2024-04-02T04:39:24.604287Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-02T04:42:12.452671Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-02T04:42:12.446025Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-02T04:35:42.664861Z", "last_refresh": "2024-04-02T04:39:24.604079Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi053:172.21.15.53=smithi053", "smithi122:172.21.15.122=smithi122"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-02T04:36:30.245267Z", "last_refresh": "2024-04-02T04:39:24.603934Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-02T04:35:48.479759Z", "last_refresh": "2024-04-02T04:39:24.604350Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-02T04:39:24.604476Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-02T04:35:45.496079Z", "last_refresh": "2024-04-02T04:39:24.604413Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-02T04:42:16.732 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:16 smithi122 ceph-mon[27145]: pgmap v214: 65 pgs: 20 creating+peering, 9 unknown, 36 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:42:16.769 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:16 smithi053 ceph-mon[24889]: pgmap v214: 65 pgs: 20 creating+peering, 9 unknown, 36 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:42:16.951 INFO:tasks.cephadm:mds.foofs has 0/2 2024-04-02T04:42:17.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:17 smithi122 ceph-mon[27145]: from='client.14474 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:42:17.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:17 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:17.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:17 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi122.mwoyoh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T04:42:17.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:17 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi122.mwoyoh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T04:42:17.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:17 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:17.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:17 smithi122 ceph-mon[27145]: mds.? [v2:172.21.15.53:6834/2480061460,v1:172.21.15.53:6835/2480061460] up:boot 2024-04-02T04:42:17.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:17 smithi122 ceph-mon[27145]: daemon mds.foofs.smithi053.nfrfot assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-04-02T04:42:17.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:17 smithi122 ceph-mon[27145]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-02T04:42:17.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:17 smithi122 ceph-mon[27145]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-02T04:42:17.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:17 smithi122 ceph-mon[27145]: Cluster is now healthy 2024-04-02T04:42:17.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:17 smithi122 ceph-mon[27145]: fsmap foofs:0 1 up:standby 2024-04-02T04:42:17.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:17 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi053.nfrfot"}]: dispatch 2024-04-02T04:42:17.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:17 smithi122 ceph-mon[27145]: fsmap foofs:1 {0=foofs.smithi053.nfrfot=up:creating} 2024-04-02T04:42:17.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:17 smithi122 ceph-mon[27145]: daemon mds.foofs.smithi053.nfrfot is now active in filesystem foofs as rank 0 2024-04-02T04:42:17.810 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:17 smithi053 ceph-mon[24889]: from='client.14474 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:42:17.810 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:17 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:17.811 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:17 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi122.mwoyoh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T04:42:17.811 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:17 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi122.mwoyoh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T04:42:17.811 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:17 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:17.811 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:17 smithi053 ceph-mon[24889]: mds.? [v2:172.21.15.53:6834/2480061460,v1:172.21.15.53:6835/2480061460] up:boot 2024-04-02T04:42:17.811 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:17 smithi053 ceph-mon[24889]: daemon mds.foofs.smithi053.nfrfot assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-04-02T04:42:17.811 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:17 smithi053 ceph-mon[24889]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-02T04:42:17.812 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:17 smithi053 ceph-mon[24889]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-02T04:42:17.812 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:17 smithi053 ceph-mon[24889]: Cluster is now healthy 2024-04-02T04:42:17.812 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:17 smithi053 ceph-mon[24889]: fsmap foofs:0 1 up:standby 2024-04-02T04:42:17.812 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:17 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi053.nfrfot"}]: dispatch 2024-04-02T04:42:17.812 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:17 smithi053 ceph-mon[24889]: fsmap foofs:1 {0=foofs.smithi053.nfrfot=up:creating} 2024-04-02T04:42:17.812 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:17 smithi053 ceph-mon[24889]: daemon mds.foofs.smithi053.nfrfot is now active in filesystem foofs as rank 0 2024-04-02T04:42:17.952 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph orch ls -f json 2024-04-02T04:42:18.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:18 smithi053 ceph-mon[24889]: Deploying daemon mds.foofs.smithi122.mwoyoh on smithi122 2024-04-02T04:42:18.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:18 smithi053 ceph-mon[24889]: pgmap v215: 65 pgs: 2 creating+peering, 63 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 201 B/s wr, 0 op/s 2024-04-02T04:42:18.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:18 smithi053 ceph-mon[24889]: mds.? [v2:172.21.15.53:6834/2480061460,v1:172.21.15.53:6835/2480061460] up:active 2024-04-02T04:42:18.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:18 smithi053 ceph-mon[24889]: fsmap foofs:1 {0=foofs.smithi053.nfrfot=up:active} 2024-04-02T04:42:18.835 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:18 smithi122 ceph-mon[27145]: Deploying daemon mds.foofs.smithi122.mwoyoh on smithi122 2024-04-02T04:42:18.835 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:18 smithi122 ceph-mon[27145]: pgmap v215: 65 pgs: 2 creating+peering, 63 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 201 B/s wr, 0 op/s 2024-04-02T04:42:18.835 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:18 smithi122 ceph-mon[27145]: mds.? [v2:172.21.15.53:6834/2480061460,v1:172.21.15.53:6835/2480061460] up:active 2024-04-02T04:42:18.835 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:18 smithi122 ceph-mon[27145]: fsmap foofs:1 {0=foofs.smithi053.nfrfot=up:active} 2024-04-02T04:42:19.892 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-02T04:42:19.893 INFO:teuthology.orchestra.run.smithi053.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-02T04:35:49.645669Z", "last_refresh": "2024-04-02T04:39:24.604155Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-02T04:35:44.306903Z", "last_refresh": "2024-04-02T04:39:24.604223Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-02T04:35:47.354203Z", "last_refresh": "2024-04-02T04:39:24.604287Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-02T04:42:12.452671Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-02T04:42:12.446025Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-02T04:35:42.664861Z", "last_refresh": "2024-04-02T04:39:24.604079Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi053:172.21.15.53=smithi053", "smithi122:172.21.15.122=smithi122"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-02T04:36:30.245267Z", "last_refresh": "2024-04-02T04:39:24.603934Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-02T04:35:48.479759Z", "last_refresh": "2024-04-02T04:39:24.604350Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-02T04:39:24.604476Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-02T04:35:45.496079Z", "last_refresh": "2024-04-02T04:39:24.604413Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-02T04:42:20.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:20 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:20.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:20 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:42:20.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:20 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:20.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:20 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:42:20.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:20 smithi053 ceph-mon[24889]: pgmap v216: 65 pgs: 65 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 579 B/s wr, 3 op/s 2024-04-02T04:42:20.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:20 smithi053 ceph-mon[24889]: mds.? [v2:172.21.15.122:6832/233414887,v1:172.21.15.122:6833/233414887] up:boot 2024-04-02T04:42:20.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:20 smithi053 ceph-mon[24889]: fsmap foofs:1 {0=foofs.smithi053.nfrfot=up:active} 1 up:standby 2024-04-02T04:42:20.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:20 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi122.mwoyoh"}]: dispatch 2024-04-02T04:42:20.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:20 smithi053 ceph-mon[24889]: fsmap foofs:1 {0=foofs.smithi053.nfrfot=up:active} 1 up:standby 2024-04-02T04:42:20.359 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:20 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:20.359 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:20 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:42:20.360 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:20 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:20.360 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:20 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:42:20.360 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:20 smithi122 ceph-mon[27145]: pgmap v216: 65 pgs: 65 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 579 B/s wr, 3 op/s 2024-04-02T04:42:20.360 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:20 smithi122 ceph-mon[27145]: mds.? [v2:172.21.15.122:6832/233414887,v1:172.21.15.122:6833/233414887] up:boot 2024-04-02T04:42:20.360 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:20 smithi122 ceph-mon[27145]: fsmap foofs:1 {0=foofs.smithi053.nfrfot=up:active} 1 up:standby 2024-04-02T04:42:20.360 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:20 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi122.mwoyoh"}]: dispatch 2024-04-02T04:42:20.360 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:20 smithi122 ceph-mon[27145]: fsmap foofs:1 {0=foofs.smithi053.nfrfot=up:active} 1 up:standby 2024-04-02T04:42:21.111 INFO:tasks.cephadm:mds.foofs has 0/2 2024-04-02T04:42:21.323 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:21 smithi122 ceph-mon[27145]: from='client.14482 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:42:21.323 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:21 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:21.333 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:21 smithi053 ceph-mon[24889]: from='client.14482 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:42:21.333 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:21 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:22.112 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph orch ls -f json 2024-04-02T04:42:22.806 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:22 smithi053 ceph-mon[24889]: pgmap v217: 65 pgs: 65 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 634 B/s wr, 3 op/s 2024-04-02T04:42:22.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:22 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:22.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:22 smithi122 ceph-mon[27145]: pgmap v217: 65 pgs: 65 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 634 B/s wr, 3 op/s 2024-04-02T04:42:22.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:22 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:24.155 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:23 smithi053 ceph-mon[24889]: pgmap v218: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-02T04:42:24.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:23 smithi122 ceph-mon[27145]: pgmap v218: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-02T04:42:24.548 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-02T04:42:24.548 INFO:teuthology.orchestra.run.smithi053.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-02T04:35:49.645669Z", "last_refresh": "2024-04-02T04:39:24.604155Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-02T04:35:44.306903Z", "last_refresh": "2024-04-02T04:39:24.604223Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-02T04:35:47.354203Z", "last_refresh": "2024-04-02T04:39:24.604287Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-02T04:42:12.452671Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-02T04:42:12.446025Z", "last_refresh": "2024-04-02T04:42:20.973986Z", "running": 1, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-02T04:35:42.664861Z", "last_refresh": "2024-04-02T04:39:24.604079Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi053:172.21.15.53=smithi053", "smithi122:172.21.15.122=smithi122"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-02T04:36:30.245267Z", "last_refresh": "2024-04-02T04:39:24.603934Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-02T04:35:48.479759Z", "last_refresh": "2024-04-02T04:39:24.604350Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-02T04:39:24.604476Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-02T04:35:45.496079Z", "last_refresh": "2024-04-02T04:39:24.604413Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-02T04:42:25.275 INFO:tasks.cephadm:mds.foofs has 1/2 2024-04-02T04:42:25.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:25 smithi122 ceph-mon[27145]: from='client.14486 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:42:25.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:25 smithi122 ceph-mon[27145]: pgmap v219: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-04-02T04:42:25.913 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:25 smithi053 ceph-mon[24889]: from='client.14486 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:42:25.913 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:25 smithi053 ceph-mon[24889]: pgmap v219: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-04-02T04:42:26.275 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph orch ls -f json 2024-04-02T04:42:27.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:27 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:27.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:27 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:27.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:27 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:27.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:27 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:42:27.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:27 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:27.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:27 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:42:27.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:27 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:27.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:27 smithi053 ceph-mon[24889]: pgmap v220: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-02T04:42:27.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:27 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:27.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:27 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:27.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:27 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:27.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:27 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:42:27.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:27 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:27.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:27 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:42:27.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:27 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:27.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:27 smithi122 ceph-mon[27145]: pgmap v220: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-02T04:42:28.303 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-02T04:42:28.303 INFO:teuthology.orchestra.run.smithi053.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-02T04:35:49.645669Z", "last_refresh": "2024-04-02T04:42:26.538302Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-02T04:35:44.306903Z", "last_refresh": "2024-04-02T04:42:20.973339Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-02T04:35:47.354203Z", "last_refresh": "2024-04-02T04:42:26.538448Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-02T04:42:12.452671Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-02T04:42:12.446025Z", "last_refresh": "2024-04-02T04:42:20.973986Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-02T04:35:42.664861Z", "last_refresh": "2024-04-02T04:42:20.973482Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi053:172.21.15.53=smithi053", "smithi122:172.21.15.122=smithi122"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-02T04:36:30.245267Z", "last_refresh": "2024-04-02T04:42:20.973560Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-02T04:35:48.479759Z", "last_refresh": "2024-04-02T04:42:20.973629Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-02T04:42:20.973698Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-02T04:35:45.496079Z", "last_refresh": "2024-04-02T04:42:26.538574Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-02T04:42:29.042 INFO:tasks.cephadm:mds.foofs has 2/2 2024-04-02T04:42:29.042 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-02T04:42:29.051 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi053.front.sepia.ceph.com 2024-04-02T04:42:29.052 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- bash -c 'ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2' 2024-04-02T04:42:29.306 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:28 smithi053 ceph-mon[24889]: from='client.14490 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:42:29.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:28 smithi122 ceph-mon[27145]: from='client.14490 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:42:30.306 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:29 smithi053 ceph-mon[24889]: pgmap v221: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1023 B/s wr, 3 op/s 2024-04-02T04:42:30.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:29 smithi122 ceph-mon[27145]: pgmap v221: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1023 B/s wr, 3 op/s 2024-04-02T04:42:31.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:31 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-04-02T04:42:31.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:31 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-04-02T04:42:32.507 INFO:teuthology.orchestra.run.smithi053.stdout:NFS Cluster Created Successfully 2024-04-02T04:42:32.753 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:32 smithi053 ceph-mon[24889]: from='client.14494 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:32.753 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:32 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-04-02T04:42:32.753 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:32 smithi053 ceph-mon[24889]: osdmap e65: 8 total, 8 up, 8 in 2024-04-02T04:42:32.753 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:32 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-04-02T04:42:32.753 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:32 smithi053 ceph-mon[24889]: pgmap v223: 97 pgs: 32 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 818 B/s wr, 1 op/s 2024-04-02T04:42:32.754 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:32 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-04-02T04:42:32.754 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:32 smithi053 ceph-mon[24889]: osdmap e66: 8 total, 8 up, 8 in 2024-04-02T04:42:32.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:32 smithi122 ceph-mon[27145]: from='client.14494 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:32.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:32 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-04-02T04:42:32.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:32 smithi122 ceph-mon[27145]: osdmap e65: 8 total, 8 up, 8 in 2024-04-02T04:42:32.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:32 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-04-02T04:42:32.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:32 smithi122 ceph-mon[27145]: pgmap v223: 97 pgs: 32 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 818 B/s wr, 1 op/s 2024-04-02T04:42:32.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:32 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-04-02T04:42:32.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:32 smithi122 ceph-mon[27145]: osdmap e66: 8 total, 8 up, 8 in 2024-04-02T04:42:33.224 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- bash -c 'ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake' 2024-04-02T04:42:33.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:33 smithi053 ceph-mon[24889]: Saving service nfs.foo spec with placement count:2 2024-04-02T04:42:33.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:33 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:33.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:33 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:42:33.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:33 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:33.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:33 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:42:33.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:33 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:33.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:33 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:33.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:33 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi053.mhkdcb", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-02T04:42:33.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:33 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi053.mhkdcb", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-02T04:42:33.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:33 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-02T04:42:33.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:33 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-02T04:42:33.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:33 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:33.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:33 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-02T04:42:33.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:33 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-02T04:42:33.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:33 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi053.mhkdcb-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-02T04:42:33.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:33 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi053.mhkdcb-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-02T04:42:33.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:33 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:33.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:33 smithi053 ceph-mon[24889]: osdmap e67: 8 total, 8 up, 8 in 2024-04-02T04:42:33.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:33 smithi122 ceph-mon[27145]: Saving service nfs.foo spec with placement count:2 2024-04-02T04:42:33.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:33 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:33.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:33 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:42:33.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:33 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:33.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:33 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:42:33.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:33 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:33.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:33 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:33.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:33 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi053.mhkdcb", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-02T04:42:33.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:33 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi053.mhkdcb", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-02T04:42:33.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:33 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-02T04:42:33.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:33 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-02T04:42:33.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:33 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:33.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:33 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-02T04:42:33.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:33 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-02T04:42:33.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:33 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi053.mhkdcb-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-02T04:42:33.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:33 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi053.mhkdcb-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-02T04:42:33.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:33 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:33.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:33 smithi122 ceph-mon[27145]: osdmap e67: 8 total, 8 up, 8 in 2024-04-02T04:42:34.777 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:34 smithi053 ceph-mon[24889]: Creating key for client.nfs.foo.0.0.smithi053.mhkdcb 2024-04-02T04:42:34.777 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:34 smithi053 ceph-mon[24889]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-04-02T04:42:34.778 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:34 smithi053 ceph-mon[24889]: Rados config object exists: conf-nfs.foo 2024-04-02T04:42:34.778 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:34 smithi053 ceph-mon[24889]: Creating key for client.nfs.foo.0.0.smithi053.mhkdcb-rgw 2024-04-02T04:42:34.778 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:34 smithi053 ceph-mon[24889]: Deploying daemon nfs.foo.0.0.smithi053.mhkdcb on smithi053 2024-04-02T04:42:34.778 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:34 smithi053 ceph-mon[24889]: pgmap v226: 97 pgs: 10 creating+peering, 18 unknown, 69 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:42:34.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:34 smithi122 ceph-mon[27145]: Creating key for client.nfs.foo.0.0.smithi053.mhkdcb 2024-04-02T04:42:34.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:34 smithi122 ceph-mon[27145]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-04-02T04:42:34.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:34 smithi122 ceph-mon[27145]: Rados config object exists: conf-nfs.foo 2024-04-02T04:42:34.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:34 smithi122 ceph-mon[27145]: Creating key for client.nfs.foo.0.0.smithi053.mhkdcb-rgw 2024-04-02T04:42:34.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:34 smithi122 ceph-mon[27145]: Deploying daemon nfs.foo.0.0.smithi053.mhkdcb on smithi053 2024-04-02T04:42:34.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:34 smithi122 ceph-mon[27145]: pgmap v226: 97 pgs: 10 creating+peering, 18 unknown, 69 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-04-02T04:42:35.744 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:35 smithi053 ceph-mon[24889]: mgrmap e20: smithi053.legdmv(active, since 5m), standbys: smithi122.cpwlyk 2024-04-02T04:42:35.744 INFO:teuthology.orchestra.run.smithi053.stderr:Invalid command: Unexpected argument '--clusterid' 2024-04-02T04:42:35.744 INFO:teuthology.orchestra.run.smithi053.stderr:nfs export create cephfs [] [--readonly] [--client_addr ...] [--squash ] : Create a CephFS export 2024-04-02T04:42:35.745 INFO:teuthology.orchestra.run.smithi053.stderr:Error EINVAL: invalid command 2024-04-02T04:42:35.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:35 smithi122 ceph-mon[27145]: mgrmap e20: smithi053.legdmv(active, since 5m), standbys: smithi122.cpwlyk 2024-04-02T04:42:36.101 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:42:36.102 INFO:teuthology.orchestra.run.smithi053.stdout: "bind": "/fake", 2024-04-02T04:42:36.102 INFO:teuthology.orchestra.run.smithi053.stdout: "fs": "foofs", 2024-04-02T04:42:36.102 INFO:teuthology.orchestra.run.smithi053.stdout: "path": "/", 2024-04-02T04:42:36.102 INFO:teuthology.orchestra.run.smithi053.stdout: "cluster": "foo", 2024-04-02T04:42:36.102 INFO:teuthology.orchestra.run.smithi053.stdout: "mode": "RW" 2024-04-02T04:42:36.102 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:42:36.758 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:36 smithi053 ceph-mon[24889]: pgmap v227: 97 pgs: 10 creating+peering, 87 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-04-02T04:42:36.758 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:36 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-04-02T04:42:36.758 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:36 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-04-02T04:42:36.780 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- bash -c 'while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done' 2024-04-02T04:42:36.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:36 smithi122 ceph-mon[27145]: pgmap v227: 97 pgs: 10 creating+peering, 87 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-04-02T04:42:36.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:36 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-04-02T04:42:36.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:36 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-04-02T04:42:37.648 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:37 smithi053 ceph-mon[24889]: from='client.14514 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:37.648 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:37 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:37.648 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:37 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi122.mauvsb", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-02T04:42:37.648 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:37 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi122.mauvsb", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-02T04:42:37.648 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:37 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-02T04:42:37.648 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:37 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-02T04:42:37.648 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:37 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:37.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:37 smithi122 ceph-mon[27145]: from='client.14514 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:37.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:37 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:37.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:37 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi122.mauvsb", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-02T04:42:37.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:37 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi122.mauvsb", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-02T04:42:37.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:37 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-02T04:42:37.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:37 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-02T04:42:37.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:37 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:38.805 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:38 smithi053 ceph-mon[24889]: Creating key for client.nfs.foo.1.0.smithi122.mauvsb 2024-04-02T04:42:38.805 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:38 smithi053 ceph-mon[24889]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-04-02T04:42:38.805 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:38 smithi053 ceph-mon[24889]: pgmap v228: 97 pgs: 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 169 B/s rd, 338 B/s wr, 0 op/s 2024-04-02T04:42:38.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:38 smithi122 ceph-mon[27145]: Creating key for client.nfs.foo.1.0.smithi122.mauvsb 2024-04-02T04:42:38.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:38 smithi122 ceph-mon[27145]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-04-02T04:42:38.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:38 smithi122 ceph-mon[27145]: pgmap v228: 97 pgs: 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 169 B/s rd, 338 B/s wr, 0 op/s 2024-04-02T04:42:40.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:40 smithi053 ceph-mon[24889]: from='client.14530 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:40.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:40 smithi053 ceph-mon[24889]: pgmap v229: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-04-02T04:42:40.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:40 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-02T04:42:40.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:40 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-02T04:42:40.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:40 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi122.mauvsb-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-02T04:42:40.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:40 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi122.mauvsb-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-02T04:42:40.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:40 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:40.856 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:40 smithi122 ceph-mon[27145]: from='client.14530 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:40.857 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:40 smithi122 ceph-mon[27145]: pgmap v229: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-04-02T04:42:40.857 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:40 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-02T04:42:40.857 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:40 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-02T04:42:40.857 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:40 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi122.mauvsb-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-02T04:42:40.857 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:40 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi122.mauvsb-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-02T04:42:40.857 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:40 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:41.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:41 smithi053 ceph-mon[24889]: Rados config object exists: conf-nfs.foo 2024-04-02T04:42:41.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:41 smithi053 ceph-mon[24889]: Creating key for client.nfs.foo.1.0.smithi122.mauvsb-rgw 2024-04-02T04:42:41.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:41 smithi053 ceph-mon[24889]: Deploying daemon nfs.foo.1.0.smithi122.mauvsb on smithi122 2024-04-02T04:42:41.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:41 smithi053 ceph-mon[24889]: from='client.14534 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:41.890 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:41 smithi122 ceph-mon[27145]: Rados config object exists: conf-nfs.foo 2024-04-02T04:42:41.890 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:41 smithi122 ceph-mon[27145]: Creating key for client.nfs.foo.1.0.smithi122.mauvsb-rgw 2024-04-02T04:42:41.890 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:41 smithi122 ceph-mon[27145]: Deploying daemon nfs.foo.1.0.smithi122.mauvsb on smithi122 2024-04-02T04:42:41.891 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:41 smithi122 ceph-mon[27145]: from='client.14534 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:42.741 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:42 smithi122 ceph-mon[27145]: pgmap v230: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-02T04:42:42.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:42 smithi053 ceph-mon[24889]: pgmap v230: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-02T04:42:43.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:43 smithi122 ceph-mon[27145]: from='client.14548 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:43.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:43 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:43.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:43 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:42:43.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:43 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:43.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:43 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:42:43.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:43 smithi053 ceph-mon[24889]: from='client.14548 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:43.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:43 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:43.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:43 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:42:43.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:43 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:43.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:43 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:42:44.806 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:44 smithi053 ceph-mon[24889]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:44.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:44 smithi053 ceph-mon[24889]: pgmap v231: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-02T04:42:44.852 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:44 smithi122 ceph-mon[27145]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:44.852 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:44 smithi122 ceph-mon[27145]: pgmap v231: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-02T04:42:45.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:45 smithi053 ceph-mon[24889]: from='client.14560 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:45.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:45 smithi122 ceph-mon[27145]: from='client.14560 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:46.806 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:46 smithi053 ceph-mon[24889]: pgmap v232: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.9 KiB/s wr, 5 op/s 2024-04-02T04:42:46.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:46 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:46.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:46 smithi122 ceph-mon[27145]: pgmap v232: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.9 KiB/s wr, 5 op/s 2024-04-02T04:42:46.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:46 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:47.806 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:47 smithi053 ceph-mon[24889]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:47.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:47 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:47.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:47 smithi122 ceph-mon[27145]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:47.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:47 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:48.362 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo ?:2049 2/2 2s ago 15s count:2 2024-04-02T04:42:48.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:48 smithi053 ceph-mon[24889]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:48.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:48 smithi053 ceph-mon[24889]: pgmap v233: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-04-02T04:42:48.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:48 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:48.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:48 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:48.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:48 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:48.704 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:48 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:42:48.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:48 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:48.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:48 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:42:48.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:48 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:48.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:48 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:48.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:48 smithi122 ceph-mon[27145]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:48.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:48 smithi122 ceph-mon[27145]: pgmap v233: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-04-02T04:42:48.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:48 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:48.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:48 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:48.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:48 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:48.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:48 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:42:48.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:48 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:42:48.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:48 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:42:48.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:48 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:48.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:48 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:42:49.129 INFO:teuthology.run_tasks:Running task vip.exec... 2024-04-02T04:42:49.190 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi053.front.sepia.ceph.com 2024-04-02T04:42:49.190 DEBUG:teuthology.orchestra.run.smithi053:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mkdir /mnt/foo' 2024-04-02T04:42:49.219 INFO:teuthology.orchestra.run.smithi053.stderr:+ mkdir /mnt/foo 2024-04-02T04:42:49.262 DEBUG:teuthology.orchestra.run.smithi053:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done' 2024-04-02T04:42:49.331 INFO:teuthology.orchestra.run.smithi053.stderr:++ hostname 2024-04-02T04:42:49.331 INFO:teuthology.orchestra.run.smithi053.stderr:+ mount -t nfs smithi053:/fake /mnt/foo -o sync 2024-04-02T04:42:49.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:49 smithi053 ceph-mon[24889]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:49.727 DEBUG:teuthology.orchestra.run.smithi053:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'echo test > /mnt/foo/testfile' 2024-04-02T04:42:49.794 INFO:teuthology.orchestra.run.smithi053.stderr:+ echo test 2024-04-02T04:42:49.805 DEBUG:teuthology.orchestra.run.smithi053:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c sync 2024-04-02T04:42:49.871 INFO:teuthology.orchestra.run.smithi053.stderr:+ sync 2024-04-02T04:42:49.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:49 smithi122 ceph-mon[27145]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:42:50.100 INFO:teuthology.run_tasks:Running task parallel... 2024-04-02T04:42:50.115 INFO:teuthology.task.parallel:starting parallel... 2024-04-02T04:42:50.115 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-02T04:42:50.116 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-02T04:42:50.116 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi053.front.sepia.ceph.com 2024-04-02T04:42:50.116 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -e sha1=6cec793d9073f0594e2364b9b687ecaf22424cca -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-02T04:42:50.117 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-02T04:42:50.117 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-02T04:42:50.117 INFO:teuthology.task.exec:Executing custom commands... 2024-04-02T04:42:50.118 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi053.front.sepia.ceph.com 2024-04-02T04:42:50.118 DEBUG:teuthology.orchestra.run.smithi053:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 600 || true' 2024-04-02T04:42:50.177 INFO:teuthology.orchestra.run.smithi053.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-02T04:42:50.177 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-02T04:42:50.177 INFO:teuthology.orchestra.run.smithi053.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-04-02T04:42:50.185 INFO:teuthology.orchestra.run.smithi053.stdout:4 of 5 processes prepared for launch 0 sec 2024-04-02T04:42:50.286 INFO:teuthology.orchestra.run.smithi053.stdout:5 of 5 processes prepared for launch 0 sec 2024-04-02T04:42:50.286 INFO:teuthology.orchestra.run.smithi053.stdout:releasing clients 2024-04-02T04:42:50.617 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:50 smithi053 ceph-mon[24889]: pgmap v234: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-04-02T04:42:50.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:50 smithi122 ceph-mon[27145]: pgmap v234: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.1 KiB/s wr, 5 op/s 2024-04-02T04:42:51.287 INFO:teuthology.orchestra.run.smithi053.stdout: 5 117 26.62 MB/sec warmup 1 sec latency 46.728 ms 2024-04-02T04:42:51.741 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:51 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 2]}]: dispatch 2024-04-02T04:42:51.741 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:51 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-04-02T04:42:51.741 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:51 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-04-02T04:42:51.741 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:51 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-04-02T04:42:51.741 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:51 smithi053 ceph-mon[24889]: pgmap v235: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-04-02T04:42:51.741 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:51 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:42:51.741 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:51 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/trash_purge_schedule"}]: dispatch 2024-04-02T04:42:51.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:51 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 2]}]: dispatch 2024-04-02T04:42:51.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:51 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-04-02T04:42:51.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:51 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-04-02T04:42:51.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:51 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-04-02T04:42:51.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:51 smithi122 ceph-mon[27145]: pgmap v235: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-04-02T04:42:51.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:51 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:42:51.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:51 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/trash_purge_schedule"}]: dispatch 2024-04-02T04:42:52.287 INFO:teuthology.orchestra.run.smithi053.stdout: 5 264 24.82 MB/sec warmup 2 sec latency 26.716 ms 2024-04-02T04:42:52.753 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:52 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 2]}]': finished 2024-04-02T04:42:52.753 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:52 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-04-02T04:42:52.754 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:52 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-04-02T04:42:52.754 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:52 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-04-02T04:42:52.754 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:52 smithi053 ceph-mon[24889]: osdmap e68: 8 total, 8 up, 8 in 2024-04-02T04:42:52.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:52 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 2]}]': finished 2024-04-02T04:42:52.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:52 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-04-02T04:42:52.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:52 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-04-02T04:42:52.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:52 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-04-02T04:42:52.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:52 smithi122 ceph-mon[27145]: osdmap e68: 8 total, 8 up, 8 in 2024-04-02T04:42:53.288 INFO:teuthology.orchestra.run.smithi053.stdout: 5 424 24.91 MB/sec warmup 3 sec latency 38.987 ms 2024-04-02T04:42:53.502 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -e sha1=6cec793d9073f0594e2364b9b687ecaf22424cca -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-02T04:42:53.785 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:53 smithi053 ceph-mon[24889]: osdmap e69: 8 total, 8 up, 8 in 2024-04-02T04:42:53.786 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:53 smithi053 ceph-mon[24889]: pgmap v238: 97 pgs: 1 active+recovering, 96 active+clean; 28 MiB data, 65 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 3.5 MiB/s wr, 216 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-02T04:42:53.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:53 smithi122 ceph-mon[27145]: osdmap e69: 8 total, 8 up, 8 in 2024-04-02T04:42:53.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:53 smithi122 ceph-mon[27145]: pgmap v238: 97 pgs: 1 active+recovering, 96 active+clean; 28 MiB data, 65 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 3.5 MiB/s wr, 216 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-02T04:42:54.288 INFO:teuthology.orchestra.run.smithi053.stdout: 5 575 24.74 MB/sec warmup 4 sec latency 39.014 ms 2024-04-02T04:42:55.288 INFO:teuthology.orchestra.run.smithi053.stdout: 5 959 21.84 MB/sec warmup 5 sec latency 38.868 ms 2024-04-02T04:42:55.687 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:55 smithi053 ceph-mon[24889]: pgmap v239: 97 pgs: 1 active+recovering, 96 active+clean; 71 MiB data, 112 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 9.0 MiB/s wr, 380 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-02T04:42:55.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:55 smithi122 ceph-mon[27145]: pgmap v239: 97 pgs: 1 active+recovering, 96 active+clean; 71 MiB data, 112 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 9.0 MiB/s wr, 380 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-02T04:42:56.288 INFO:teuthology.orchestra.run.smithi053.stdout: 5 1526 21.73 MB/sec warmup 6 sec latency 44.876 ms 2024-04-02T04:42:57.288 INFO:teuthology.orchestra.run.smithi053.stdout: 5 2303 19.65 MB/sec warmup 7 sec latency 35.232 ms 2024-04-02T04:42:57.695 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:57 smithi053 ceph-mon[24889]: pgmap v240: 97 pgs: 1 active+recovering, 96 active+clean; 104 MiB data, 143 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 14 MiB/s wr, 509 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-02T04:42:57.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:57 smithi122 ceph-mon[27145]: pgmap v240: 97 pgs: 1 active+recovering, 96 active+clean; 104 MiB data, 143 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 14 MiB/s wr, 509 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-02T04:42:58.055 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -e sha1=6cec793d9073f0594e2364b9b687ecaf22424cca -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-02T04:42:58.288 INFO:teuthology.orchestra.run.smithi053.stdout: 5 2899 20.52 MB/sec warmup 8 sec latency 50.741 ms 2024-04-02T04:42:59.288 INFO:teuthology.orchestra.run.smithi053.stdout: 5 3677 21.50 MB/sec warmup 9 sec latency 20.603 ms 2024-04-02T04:42:59.713 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:42:59 smithi053 ceph-mon[24889]: pgmap v241: 97 pgs: 97 active+clean; 119 MiB data, 184 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 16 MiB/s wr, 862 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-02T04:42:59.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:42:59 smithi122 ceph-mon[27145]: pgmap v241: 97 pgs: 97 active+clean; 119 MiB data, 184 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 16 MiB/s wr, 862 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-02T04:43:00.288 INFO:teuthology.orchestra.run.smithi053.stdout: 5 4043 20.03 MB/sec warmup 10 sec latency 22.556 ms 2024-04-02T04:43:01.288 INFO:teuthology.orchestra.run.smithi053.stdout: 5 4748 19.15 MB/sec warmup 11 sec latency 24.514 ms 2024-04-02T04:43:01.738 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:01 smithi053 ceph-mon[24889]: pgmap v242: 97 pgs: 97 active+clean; 121 MiB data, 200 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 14 MiB/s wr, 711 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-02T04:43:01.897 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:01 smithi122 ceph-mon[27145]: pgmap v242: 97 pgs: 97 active+clean; 121 MiB data, 200 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 14 MiB/s wr, 711 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-02T04:43:02.288 INFO:teuthology.orchestra.run.smithi053.stdout: 5 5503 18.99 MB/sec warmup 12 sec latency 32.809 ms 2024-04-02T04:43:03.128 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -e sha1=6cec793d9073f0594e2364b9b687ecaf22424cca -- bash -c 'ceph mgr module enable nfs --force' 2024-04-02T04:43:03.288 INFO:teuthology.orchestra.run.smithi053.stdout: 5 6118 18.66 MB/sec warmup 13 sec latency 44.846 ms 2024-04-02T04:43:04.012 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:03 smithi053 ceph-mon[24889]: pgmap v243: 97 pgs: 97 active+clean; 133 MiB data, 263 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 14 MiB/s wr, 881 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-02T04:43:04.288 INFO:teuthology.orchestra.run.smithi053.stdout: 5 6953 19.70 MB/sec warmup 14 sec latency 20.762 ms 2024-04-02T04:43:04.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:03 smithi122 ceph-mon[27145]: pgmap v243: 97 pgs: 97 active+clean; 133 MiB data, 263 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 14 MiB/s wr, 881 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-02T04:43:05.288 INFO:teuthology.orchestra.run.smithi053.stdout: 5 7536 19.73 MB/sec warmup 15 sec latency 21.930 ms 2024-04-02T04:43:05.775 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:05 smithi053 ceph-mon[24889]: pgmap v244: 97 pgs: 97 active+clean; 134 MiB data, 299 MiB used, 715 GiB / 715 GiB avail; 6.7 KiB/s rd, 13 MiB/s wr, 850 op/s 2024-04-02T04:43:05.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:05 smithi122 ceph-mon[27145]: pgmap v244: 97 pgs: 97 active+clean; 134 MiB data, 299 MiB used, 715 GiB / 715 GiB avail; 6.7 KiB/s rd, 13 MiB/s wr, 850 op/s 2024-04-02T04:43:06.288 INFO:teuthology.orchestra.run.smithi053.stdout: 5 8030 18.79 MB/sec warmup 16 sec latency 60.789 ms 2024-04-02T04:43:06.724 INFO:teuthology.orchestra.run.smithi053.stderr:module 'nfs' is already enabled 2024-04-02T04:43:06.797 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:06 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/1151896551' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-04-02T04:43:06.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:06 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/1151896551' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-04-02T04:43:07.288 INFO:teuthology.orchestra.run.smithi053.stdout: 5 8593 18.75 MB/sec warmup 17 sec latency 30.544 ms 2024-04-02T04:43:07.812 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:07 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/1151896551' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-04-02T04:43:07.812 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:07 smithi053 ceph-mon[24889]: mgrmap e21: smithi053.legdmv(active, since 6m), standbys: smithi122.cpwlyk 2024-04-02T04:43:07.813 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:07 smithi053 ceph-mon[24889]: pgmap v245: 97 pgs: 97 active+clean; 144 MiB data, 344 MiB used, 715 GiB / 715 GiB avail; 6.6 KiB/s rd, 12 MiB/s wr, 950 op/s 2024-04-02T04:43:07.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:07 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/1151896551' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-04-02T04:43:07.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:07 smithi122 ceph-mon[27145]: mgrmap e21: smithi053.legdmv(active, since 6m), standbys: smithi122.cpwlyk 2024-04-02T04:43:07.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:07 smithi122 ceph-mon[27145]: pgmap v245: 97 pgs: 97 active+clean; 144 MiB data, 344 MiB used, 715 GiB / 715 GiB avail; 6.6 KiB/s rd, 12 MiB/s wr, 950 op/s 2024-04-02T04:43:07.990 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -e sha1=6cec793d9073f0594e2364b9b687ecaf22424cca -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-02T04:43:08.289 INFO:teuthology.orchestra.run.smithi053.stdout: 5 9294 18.12 MB/sec warmup 18 sec latency 48.165 ms 2024-04-02T04:43:09.289 INFO:teuthology.orchestra.run.smithi053.stdout: 5 9715 17.88 MB/sec warmup 19 sec latency 23.711 ms 2024-04-02T04:43:09.663 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:09 smithi053 ceph-mon[24889]: pgmap v246: 97 pgs: 97 active+clean; 143 MiB data, 397 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 9.3 MiB/s wr, 907 op/s 2024-04-02T04:43:09.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:09 smithi122 ceph-mon[27145]: pgmap v246: 97 pgs: 97 active+clean; 143 MiB data, 397 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 9.3 MiB/s wr, 907 op/s 2024-04-02T04:43:10.289 INFO:teuthology.orchestra.run.smithi053.stdout: 5 10485 18.57 MB/sec warmup 20 sec latency 28.720 ms 2024-04-02T04:43:10.887 INFO:teuthology.orchestra.run.smithi053.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca 2024-04-02T04:43:11.289 INFO:teuthology.orchestra.run.smithi053.stdout: 5 11003 18.61 MB/sec warmup 21 sec latency 24.243 ms 2024-04-02T04:43:11.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:11 smithi053 ceph-mon[24889]: from='client.14592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:43:11.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:11 smithi053 ceph-mon[24889]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca 2024-04-02T04:43:11.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:11 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:43:11.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:11 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:43:12.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:11 smithi122 ceph-mon[27145]: from='client.14592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:43:12.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:11 smithi122 ceph-mon[27145]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca 2024-04-02T04:43:12.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:11 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:43:12.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:11 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:43:12.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:11 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:43:12.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:11 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:43:12.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:11 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:43:12.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:11 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:43:12.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:11 smithi122 ceph-mon[27145]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca 2024-04-02T04:43:12.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:11 smithi122 ceph-mon[27145]: pgmap v247: 97 pgs: 97 active+clean; 142 MiB data, 402 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 7.9 MiB/s wr, 688 op/s 2024-04-02T04:43:12.185 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:11 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:43:12.185 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:11 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:43:12.185 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:11 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:43:12.185 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:11 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:43:12.185 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:11 smithi053 ceph-mon[24889]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca 2024-04-02T04:43:12.185 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:11 smithi053 ceph-mon[24889]: pgmap v247: 97 pgs: 97 active+clean; 142 MiB data, 402 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 7.9 MiB/s wr, 688 op/s 2024-04-02T04:43:12.289 INFO:teuthology.orchestra.run.smithi053.stdout: 5 11527 17.95 MB/sec warmup 22 sec latency 46.424 ms 2024-04-02T04:43:12.459 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-02T04:43:12.460 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi053.front.sepia.ceph.com 2024-04-02T04:43:12.460 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -e sha1=6cec793d9073f0594e2364b9b687ecaf22424cca -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-02T04:43:13.289 INFO:teuthology.orchestra.run.smithi053.stdout: 5 12118 18.01 MB/sec warmup 23 sec latency 29.639 ms 2024-04-02T04:43:14.225 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:13 smithi053 ceph-mon[24889]: pgmap v248: 97 pgs: 97 active+clean; 150 MiB data, 427 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 9.7 MiB/s wr, 928 op/s 2024-04-02T04:43:14.289 INFO:teuthology.orchestra.run.smithi053.stdout: 5 12816 17.54 MB/sec warmup 24 sec latency 68.464 ms 2024-04-02T04:43:14.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:13 smithi122 ceph-mon[27145]: pgmap v248: 97 pgs: 97 active+clean; 150 MiB data, 427 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 9.7 MiB/s wr, 928 op/s 2024-04-02T04:43:15.289 INFO:teuthology.orchestra.run.smithi053.stdout: 5 13175 17.21 MB/sec warmup 25 sec latency 26.797 ms 2024-04-02T04:43:15.741 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:15 smithi053 ceph-mon[24889]: pgmap v249: 97 pgs: 97 active+clean; 153 MiB data, 437 MiB used, 715 GiB / 715 GiB avail; 9.3 KiB/s rd, 8.7 MiB/s wr, 825 op/s 2024-04-02T04:43:15.878 INFO:teuthology.orchestra.run.smithi053.stdout:true 2024-04-02T04:43:15.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:15 smithi122 ceph-mon[27145]: pgmap v249: 97 pgs: 97 active+clean; 153 MiB data, 437 MiB used, 715 GiB / 715 GiB avail; 9.3 KiB/s rd, 8.7 MiB/s wr, 825 op/s 2024-04-02T04:43:16.289 INFO:teuthology.orchestra.run.smithi053.stdout: 5 13702 17.50 MB/sec warmup 26 sec latency 48.766 ms 2024-04-02T04:43:16.749 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:16 smithi053 ceph-mon[24889]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:43:16.750 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:16 smithi053 ceph-mon[24889]: from='client.14600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:43:16.800 INFO:teuthology.orchestra.run.smithi053.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-02T04:43:16.800 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager.smithi053 smithi053 *:9093,9094 running (5m) 29s ago 6m 23.3M - ba2b418f427c 8d27e10ebf84 2024-04-02T04:43:16.801 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi053 smithi053 running (6m) 29s ago 6m 6409k - 17.2.0 e1d6a67b021e a5c436bbd36d 2024-04-02T04:43:16.801 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi122 smithi122 running (6m) 31s ago 6m 6405k - 17.2.0 e1d6a67b021e b7074474cd46 2024-04-02T04:43:16.801 INFO:teuthology.orchestra.run.smithi053.stdout:grafana.smithi053 smithi053 *:3000 running (5m) 29s ago 6m 52.0M - 8.3.5 dad864ee21e9 8f4a8d871d62 2024-04-02T04:43:16.801 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi053.nfrfot smithi053 running (60s) 29s ago 60s 14.9M - 17.2.0 e1d6a67b021e c94e9e643514 2024-04-02T04:43:16.801 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi122.mwoyoh smithi122 running (58s) 31s ago 57s 13.4M - 17.2.0 e1d6a67b021e 73376d9ee4b8 2024-04-02T04:43:16.801 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi053.legdmv smithi053 *:9283 running (8m) 29s ago 8m 460M - 17.2.0 e1d6a67b021e 5616241666e4 2024-04-02T04:43:16.801 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi122.cpwlyk smithi122 *:8443,9283 running (6m) 31s ago 6m 412M - 17.2.0 e1d6a67b021e b28b3080c55d 2024-04-02T04:43:16.801 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi053 smithi053 running (8m) 29s ago 8m 47.5M 2048M 17.2.0 e1d6a67b021e 665e4412f6da 2024-04-02T04:43:16.801 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi122 smithi122 running (6m) 31s ago 6m 36.2M 2048M 17.2.0 e1d6a67b021e a2df07e0883d 2024-04-02T04:43:16.801 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.0.0.smithi053.mhkdcb smithi053 *:2049 running (39s) 29s ago 39s 46.9M - 4.0 e1d6a67b021e 07e5da7bd2c5 2024-04-02T04:43:16.801 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.1.0.smithi122.mauvsb smithi122 *:2049 running (34s) 31s ago 34s 48.7M - 4.0 e1d6a67b021e 4f866685005e 2024-04-02T04:43:16.801 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi053 smithi053 *:9100 running (6m) 29s ago 6m 16.8M - 1dbe0e931976 87185e572b31 2024-04-02T04:43:16.801 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi122 smithi122 *:9100 running (5m) 31s ago 5m 19.2M - 1dbe0e931976 d576cf080067 2024-04-02T04:43:16.801 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 smithi053 running (5m) 29s ago 5m 42.8M 3217M 17.2.0 e1d6a67b021e 906cb1fd8042 2024-04-02T04:43:16.801 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 smithi053 running (4m) 29s ago 4m 45.9M 3217M 17.2.0 e1d6a67b021e ac1ad6266e63 2024-04-02T04:43:16.802 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 smithi053 running (4m) 29s ago 4m 40.3M 3217M 17.2.0 e1d6a67b021e 86964c8a94ff 2024-04-02T04:43:16.802 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 smithi053 running (3m) 29s ago 3m 39.8M 3217M 17.2.0 e1d6a67b021e 7e8b9e98ae2b 2024-04-02T04:43:16.802 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 smithi122 running (3m) 31s ago 3m 41.0M 3984M 17.2.0 e1d6a67b021e 46076afcd7bc 2024-04-02T04:43:16.802 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 smithi122 running (3m) 31s ago 3m 41.2M 3984M 17.2.0 e1d6a67b021e e687a2f6da4d 2024-04-02T04:43:16.802 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 smithi122 running (2m) 31s ago 2m 38.8M 3984M 17.2.0 e1d6a67b021e 9e371df9a839 2024-04-02T04:43:16.802 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 smithi122 running (2m) 31s ago 2m 40.3M 3984M 17.2.0 e1d6a67b021e 9b3cc398254d 2024-04-02T04:43:16.802 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus.smithi053 smithi053 *:9095 running (5m) 29s ago 5m 62.1M - 514e6a882f6e 36d493ce400e 2024-04-02T04:43:16.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:16 smithi122 ceph-mon[27145]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:43:16.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:16 smithi122 ceph-mon[27145]: from='client.14600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:43:17.289 INFO:teuthology.orchestra.run.smithi053.stdout: 5 14157 17.63 MB/sec warmup 27 sec latency 69.732 ms 2024-04-02T04:43:17.413 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:43:17.413 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": { 2024-04-02T04:43:17.413 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:43:17.413 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:43:17.413 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": { 2024-04-02T04:43:17.414 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:43:17.414 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:43:17.414 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": { 2024-04-02T04:43:17.414 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-02T04:43:17.414 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:43:17.414 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": { 2024-04-02T04:43:17.414 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:43:17.414 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:43:17.414 INFO:teuthology.orchestra.run.smithi053.stdout: "overall": { 2024-04-02T04:43:17.414 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-04-02T04:43:17.414 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-02T04:43:17.414 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:43:17.755 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:17 smithi053 ceph-mon[24889]: from='client.24405 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:43:17.755 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:17 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/3230904560' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:43:17.755 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:17 smithi053 ceph-mon[24889]: pgmap v250: 97 pgs: 97 active+clean; 159 MiB data, 498 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9.2 MiB/s wr, 847 op/s 2024-04-02T04:43:17.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:17 smithi122 ceph-mon[27145]: from='client.24405 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:43:17.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:17 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/3230904560' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:43:17.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:17 smithi122 ceph-mon[27145]: pgmap v250: 97 pgs: 97 active+clean; 159 MiB data, 498 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9.2 MiB/s wr, 847 op/s 2024-04-02T04:43:17.973 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:43:17.973 INFO:teuthology.orchestra.run.smithi053.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca", 2024-04-02T04:43:17.973 INFO:teuthology.orchestra.run.smithi053.stdout: "in_progress": true, 2024-04-02T04:43:17.973 INFO:teuthology.orchestra.run.smithi053.stdout: "services_complete": [], 2024-04-02T04:43:17.973 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": "", 2024-04-02T04:43:17.973 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca image" 2024-04-02T04:43:17.973 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:43:18.289 INFO:teuthology.orchestra.run.smithi053.stdout: 5 14498 17.35 MB/sec warmup 28 sec latency 26.093 ms 2024-04-02T04:43:18.548 INFO:teuthology.orchestra.run.smithi053.stdout:HEALTH_OK 2024-04-02T04:43:19.023 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:18 smithi053 ceph-mon[24889]: from='client.14612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:43:19.023 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:18 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/2628586806' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:43:19.289 INFO:teuthology.orchestra.run.smithi053.stdout: 5 14984 16.87 MB/sec warmup 29 sec latency 32.879 ms 2024-04-02T04:43:19.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:18 smithi122 ceph-mon[27145]: from='client.14612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:43:19.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:18 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/2628586806' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:43:20.028 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:19 smithi053 ceph-mon[24889]: pgmap v251: 97 pgs: 97 active+clean; 163 MiB data, 530 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 7.4 MiB/s wr, 674 op/s 2024-04-02T04:43:20.290 INFO:teuthology.orchestra.run.smithi053.stdout: 5 15715 17.09 MB/sec warmup 30 sec latency 18.772 ms 2024-04-02T04:43:20.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:19 smithi122 ceph-mon[27145]: pgmap v251: 97 pgs: 97 active+clean; 163 MiB data, 530 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 7.4 MiB/s wr, 674 op/s 2024-04-02T04:43:21.290 INFO:teuthology.orchestra.run.smithi053.stdout: 5 16348 16.64 MB/sec warmup 31 sec latency 60.279 ms 2024-04-02T04:43:21.795 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:21 smithi053 ceph-mon[24889]: pgmap v252: 97 pgs: 97 active+clean; 164 MiB data, 555 MiB used, 715 GiB / 715 GiB avail; 7.9 KiB/s rd, 7.0 MiB/s wr, 638 op/s 2024-04-02T04:43:21.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:21 smithi122 ceph-mon[27145]: pgmap v252: 97 pgs: 97 active+clean; 164 MiB data, 555 MiB used, 715 GiB / 715 GiB avail; 7.9 KiB/s rd, 7.0 MiB/s wr, 638 op/s 2024-04-02T04:43:22.290 INFO:teuthology.orchestra.run.smithi053.stdout: 5 16643 16.35 MB/sec warmup 32 sec latency 25.260 ms 2024-04-02T04:43:23.290 INFO:teuthology.orchestra.run.smithi053.stdout: 5 17072 16.55 MB/sec warmup 33 sec latency 43.910 ms 2024-04-02T04:43:24.114 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:23 smithi053 ceph-mon[24889]: pgmap v253: 97 pgs: 97 active+clean; 163 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 8.5 MiB/s wr, 789 op/s 2024-04-02T04:43:24.290 INFO:teuthology.orchestra.run.smithi053.stdout: 5 17703 16.81 MB/sec warmup 34 sec latency 71.739 ms 2024-04-02T04:43:24.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:23 smithi122 ceph-mon[27145]: pgmap v253: 97 pgs: 97 active+clean; 163 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 8.5 MiB/s wr, 789 op/s 2024-04-02T04:43:25.290 INFO:teuthology.orchestra.run.smithi053.stdout: 5 18142 16.65 MB/sec warmup 35 sec latency 46.227 ms 2024-04-02T04:43:25.624 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:25 smithi053 ceph-mon[24889]: pgmap v254: 97 pgs: 97 active+clean; 167 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 9.4 KiB/s rd, 7.5 MiB/s wr, 670 op/s 2024-04-02T04:43:25.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:25 smithi122 ceph-mon[27145]: pgmap v254: 97 pgs: 97 active+clean; 167 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 9.4 KiB/s rd, 7.5 MiB/s wr, 670 op/s 2024-04-02T04:43:26.290 INFO:teuthology.orchestra.run.smithi053.stdout: 5 18613 16.28 MB/sec warmup 36 sec latency 60.099 ms 2024-04-02T04:43:27.290 INFO:teuthology.orchestra.run.smithi053.stdout: 5 18928 16.03 MB/sec warmup 37 sec latency 40.777 ms 2024-04-02T04:43:27.683 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:27 smithi053 ceph-mon[24889]: pgmap v255: 97 pgs: 97 active+clean; 178 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 7.8 MiB/s wr, 696 op/s 2024-04-02T04:43:27.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:27 smithi122 ceph-mon[27145]: pgmap v255: 97 pgs: 97 active+clean; 178 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 7.8 MiB/s wr, 696 op/s 2024-04-02T04:43:28.290 INFO:teuthology.orchestra.run.smithi053.stdout: 5 19704 16.06 MB/sec warmup 38 sec latency 27.546 ms 2024-04-02T04:43:29.290 INFO:teuthology.orchestra.run.smithi053.stdout: 5 20417 16.06 MB/sec warmup 39 sec latency 20.933 ms 2024-04-02T04:43:29.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:29 smithi053 ceph-mon[24889]: pgmap v256: 97 pgs: 97 active+clean; 181 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 7.0 MiB/s wr, 689 op/s 2024-04-02T04:43:29.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:29 smithi122 ceph-mon[27145]: pgmap v256: 97 pgs: 97 active+clean; 181 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 7.0 MiB/s wr, 689 op/s 2024-04-02T04:43:30.291 INFO:teuthology.orchestra.run.smithi053.stdout: 5 21179 16.46 MB/sec warmup 40 sec latency 21.755 ms 2024-04-02T04:43:31.291 INFO:teuthology.orchestra.run.smithi053.stdout: 5 21628 16.52 MB/sec warmup 41 sec latency 25.228 ms 2024-04-02T04:43:31.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:31 smithi053 ceph-mon[24889]: pgmap v257: 97 pgs: 97 active+clean; 181 MiB data, 687 MiB used, 715 GiB / 715 GiB avail; 9.9 KiB/s rd, 6.7 MiB/s wr, 670 op/s 2024-04-02T04:43:31.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:31 smithi122 ceph-mon[27145]: pgmap v257: 97 pgs: 97 active+clean; 181 MiB data, 687 MiB used, 715 GiB / 715 GiB avail; 9.9 KiB/s rd, 6.7 MiB/s wr, 670 op/s 2024-04-02T04:43:32.291 INFO:teuthology.orchestra.run.smithi053.stdout: 5 22065 16.20 MB/sec warmup 42 sec latency 32.658 ms 2024-04-02T04:43:33.291 INFO:teuthology.orchestra.run.smithi053.stdout: 5 22576 16.09 MB/sec warmup 43 sec latency 33.099 ms 2024-04-02T04:43:34.001 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:33 smithi053 ceph-mon[24889]: pgmap v258: 97 pgs: 97 active+clean; 186 MiB data, 725 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 8.5 MiB/s wr, 830 op/s 2024-04-02T04:43:34.291 INFO:teuthology.orchestra.run.smithi053.stdout: 5 23339 16.06 MB/sec warmup 44 sec latency 33.963 ms 2024-04-02T04:43:34.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:33 smithi122 ceph-mon[27145]: pgmap v258: 97 pgs: 97 active+clean; 186 MiB data, 725 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 8.5 MiB/s wr, 830 op/s 2024-04-02T04:43:35.291 INFO:teuthology.orchestra.run.smithi053.stdout: 5 23807 15.91 MB/sec warmup 45 sec latency 32.868 ms 2024-04-02T04:43:35.779 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:35 smithi053 ceph-mon[24889]: pgmap v259: 97 pgs: 97 active+clean; 187 MiB data, 736 MiB used, 715 GiB / 715 GiB avail; 8.6 KiB/s rd, 7.9 MiB/s wr, 756 op/s 2024-04-02T04:43:35.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:35 smithi122 ceph-mon[27145]: pgmap v259: 97 pgs: 97 active+clean; 187 MiB data, 736 MiB used, 715 GiB / 715 GiB avail; 8.6 KiB/s rd, 7.9 MiB/s wr, 756 op/s 2024-04-02T04:43:36.291 INFO:teuthology.orchestra.run.smithi053.stdout: 5 24597 16.28 MB/sec warmup 46 sec latency 35.721 ms 2024-04-02T04:43:37.291 INFO:teuthology.orchestra.run.smithi053.stdout: 5 25290 16.47 MB/sec warmup 47 sec latency 13.727 ms 2024-04-02T04:43:37.791 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:37 smithi053 ceph-mon[24889]: pgmap v260: 97 pgs: 97 active+clean; 194 MiB data, 772 MiB used, 714 GiB / 715 GiB avail; 8.4 KiB/s rd, 9.0 MiB/s wr, 853 op/s 2024-04-02T04:43:37.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:37 smithi122 ceph-mon[27145]: pgmap v260: 97 pgs: 97 active+clean; 194 MiB data, 772 MiB used, 714 GiB / 715 GiB avail; 8.4 KiB/s rd, 9.0 MiB/s wr, 853 op/s 2024-04-02T04:43:38.291 INFO:teuthology.orchestra.run.smithi053.stdout: 5 25936 16.29 MB/sec warmup 48 sec latency 10.402 ms 2024-04-02T04:43:39.291 INFO:teuthology.orchestra.run.smithi053.stdout: 5 26836 16.36 MB/sec warmup 49 sec latency 21.751 ms 2024-04-02T04:43:39.800 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:39 smithi053 ceph-mon[24889]: pgmap v261: 97 pgs: 97 active+clean; 199 MiB data, 815 MiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 8.1 MiB/s wr, 736 op/s 2024-04-02T04:43:39.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:39 smithi122 ceph-mon[27145]: pgmap v261: 97 pgs: 97 active+clean; 199 MiB data, 815 MiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 8.1 MiB/s wr, 736 op/s 2024-04-02T04:43:40.291 INFO:teuthology.orchestra.run.smithi053.stdout: 5 27737 16.61 MB/sec warmup 50 sec latency 19.201 ms 2024-04-02T04:43:41.291 INFO:teuthology.orchestra.run.smithi053.stdout: 5 28588 16.96 MB/sec warmup 51 sec latency 21.390 ms 2024-04-02T04:43:41.810 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:41 smithi053 ceph-mon[24889]: pgmap v262: 97 pgs: 97 active+clean; 204 MiB data, 823 MiB used, 714 GiB / 715 GiB avail; 8.2 KiB/s rd, 7.8 MiB/s wr, 662 op/s 2024-04-02T04:43:41.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:41 smithi122 ceph-mon[27145]: pgmap v262: 97 pgs: 97 active+clean; 204 MiB data, 823 MiB used, 714 GiB / 715 GiB avail; 8.2 KiB/s rd, 7.8 MiB/s wr, 662 op/s 2024-04-02T04:43:42.291 INFO:teuthology.orchestra.run.smithi053.stdout: 5 29090 16.75 MB/sec warmup 52 sec latency 26.052 ms 2024-04-02T04:43:43.291 INFO:teuthology.orchestra.run.smithi053.stdout: 5 29593 16.63 MB/sec warmup 53 sec latency 22.244 ms 2024-04-02T04:43:44.082 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:43 smithi053 ceph-mon[24889]: pgmap v263: 97 pgs: 97 active+clean; 217 MiB data, 876 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 11 MiB/s wr, 1k op/s 2024-04-02T04:43:44.291 INFO:teuthology.orchestra.run.smithi053.stdout: 5 30202 16.62 MB/sec warmup 54 sec latency 42.655 ms 2024-04-02T04:43:44.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:43 smithi122 ceph-mon[27145]: pgmap v263: 97 pgs: 97 active+clean; 217 MiB data, 876 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 11 MiB/s wr, 1k op/s 2024-04-02T04:43:45.291 INFO:teuthology.orchestra.run.smithi053.stdout: 5 30987 16.62 MB/sec warmup 55 sec latency 21.527 ms 2024-04-02T04:43:45.841 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:45 smithi053 ceph-mon[24889]: pgmap v264: 97 pgs: 97 active+clean; 210 MiB data, 893 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.8 MiB/s wr, 902 op/s 2024-04-02T04:43:45.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:45 smithi122 ceph-mon[27145]: pgmap v264: 97 pgs: 97 active+clean; 210 MiB data, 893 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.8 MiB/s wr, 902 op/s 2024-04-02T04:43:46.291 INFO:teuthology.orchestra.run.smithi053.stdout: 5 31714 16.87 MB/sec warmup 56 sec latency 17.494 ms 2024-04-02T04:43:47.291 INFO:teuthology.orchestra.run.smithi053.stdout: 5 32360 16.93 MB/sec warmup 57 sec latency 20.045 ms 2024-04-02T04:43:47.687 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:47 smithi053 ceph-mon[24889]: pgmap v265: 97 pgs: 97 active+clean; 213 MiB data, 918 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 884 op/s 2024-04-02T04:43:47.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:47 smithi122 ceph-mon[27145]: pgmap v265: 97 pgs: 97 active+clean; 213 MiB data, 918 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 884 op/s 2024-04-02T04:43:48.291 INFO:teuthology.orchestra.run.smithi053.stdout: 5 33064 16.81 MB/sec warmup 58 sec latency 47.684 ms 2024-04-02T04:43:49.137 INFO:teuthology.orchestra.run.smithi053.stdout:true 2024-04-02T04:43:49.291 INFO:teuthology.orchestra.run.smithi053.stdout: 5 33799 16.84 MB/sec warmup 59 sec latency 31.233 ms 2024-04-02T04:43:49.697 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:49 smithi053 ceph-mon[24889]: from='client.14620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:43:49.698 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:49 smithi053 ceph-mon[24889]: pgmap v266: 97 pgs: 97 active+clean; 226 MiB data, 998 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.8 MiB/s wr, 928 op/s 2024-04-02T04:43:49.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:49 smithi122 ceph-mon[27145]: from='client.14620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:43:49.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:49 smithi122 ceph-mon[27145]: pgmap v266: 97 pgs: 97 active+clean; 226 MiB data, 998 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.8 MiB/s wr, 928 op/s 2024-04-02T04:43:50.227 INFO:teuthology.orchestra.run.smithi053.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-02T04:43:50.227 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager.smithi053 smithi053 *:9093,9094 running (6m) 62s ago 7m 23.3M - ba2b418f427c 8d27e10ebf84 2024-04-02T04:43:50.227 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi053 smithi053 running (7m) 62s ago 7m 6409k - 17.2.0 e1d6a67b021e a5c436bbd36d 2024-04-02T04:43:50.227 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi122 smithi122 running (6m) 64s ago 6m 6405k - 17.2.0 e1d6a67b021e b7074474cd46 2024-04-02T04:43:50.228 INFO:teuthology.orchestra.run.smithi053.stdout:grafana.smithi053 smithi053 *:3000 running (5m) 62s ago 7m 52.0M - 8.3.5 dad864ee21e9 8f4a8d871d62 2024-04-02T04:43:50.228 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi053.nfrfot smithi053 running (94s) 62s ago 93s 14.9M - 17.2.0 e1d6a67b021e c94e9e643514 2024-04-02T04:43:50.228 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi122.mwoyoh smithi122 running (91s) 64s ago 91s 13.4M - 17.2.0 e1d6a67b021e 73376d9ee4b8 2024-04-02T04:43:50.228 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi053.legdmv smithi053 *:9283 running (8m) 62s ago 8m 460M - 17.2.0 e1d6a67b021e 5616241666e4 2024-04-02T04:43:50.228 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi122.cpwlyk smithi122 *:8443,9283 running (6m) 64s ago 6m 412M - 17.2.0 e1d6a67b021e b28b3080c55d 2024-04-02T04:43:50.228 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi053 smithi053 running (8m) 62s ago 8m 47.5M 2048M 17.2.0 e1d6a67b021e 665e4412f6da 2024-04-02T04:43:50.228 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi122 smithi122 running (6m) 64s ago 6m 36.2M 2048M 17.2.0 e1d6a67b021e a2df07e0883d 2024-04-02T04:43:50.228 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.0.0.smithi053.mhkdcb smithi053 *:2049 running (73s) 62s ago 72s 46.9M - 4.0 e1d6a67b021e 07e5da7bd2c5 2024-04-02T04:43:50.228 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.1.0.smithi122.mauvsb smithi122 *:2049 running (67s) 64s ago 67s 48.7M - 4.0 e1d6a67b021e 4f866685005e 2024-04-02T04:43:50.228 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi053 smithi053 *:9100 running (7m) 62s ago 7m 16.8M - 1dbe0e931976 87185e572b31 2024-04-02T04:43:50.228 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi122 smithi122 *:9100 running (6m) 64s ago 6m 19.2M - 1dbe0e931976 d576cf080067 2024-04-02T04:43:50.228 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 smithi053 running (5m) 62s ago 5m 42.8M 3217M 17.2.0 e1d6a67b021e 906cb1fd8042 2024-04-02T04:43:50.228 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 smithi053 running (5m) 62s ago 5m 45.9M 3217M 17.2.0 e1d6a67b021e ac1ad6266e63 2024-04-02T04:43:50.228 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 smithi053 running (4m) 62s ago 4m 40.3M 3217M 17.2.0 e1d6a67b021e 86964c8a94ff 2024-04-02T04:43:50.228 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 smithi053 running (4m) 62s ago 4m 39.8M 3217M 17.2.0 e1d6a67b021e 7e8b9e98ae2b 2024-04-02T04:43:50.229 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 smithi122 running (4m) 64s ago 4m 41.0M 3984M 17.2.0 e1d6a67b021e 46076afcd7bc 2024-04-02T04:43:50.229 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 smithi122 running (3m) 64s ago 3m 41.2M 3984M 17.2.0 e1d6a67b021e e687a2f6da4d 2024-04-02T04:43:50.229 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 smithi122 running (3m) 64s ago 3m 38.8M 3984M 17.2.0 e1d6a67b021e 9e371df9a839 2024-04-02T04:43:50.229 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 smithi122 running (2m) 64s ago 2m 40.3M 3984M 17.2.0 e1d6a67b021e 9b3cc398254d 2024-04-02T04:43:50.229 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus.smithi053 smithi053 *:9095 running (6m) 62s ago 6m 62.1M - 514e6a882f6e 36d493ce400e 2024-04-02T04:43:50.291 INFO:teuthology.orchestra.run.smithi053.stdout: 5 34469 16.77 MB/sec warmup 60 sec latency 38.903 ms 2024-04-02T04:43:50.701 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:50 smithi053 ceph-mon[24889]: from='client.14624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:43:50.701 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:50 smithi053 ceph-mon[24889]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:43:50.885 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:43:50.885 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": { 2024-04-02T04:43:50.885 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:43:50.885 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:43:50.885 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": { 2024-04-02T04:43:50.885 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:43:50.885 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:43:50.886 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": { 2024-04-02T04:43:50.886 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-02T04:43:50.886 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:43:50.886 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": { 2024-04-02T04:43:50.886 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:43:50.886 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:43:50.886 INFO:teuthology.orchestra.run.smithi053.stdout: "overall": { 2024-04-02T04:43:50.886 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-04-02T04:43:50.886 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-02T04:43:50.886 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:43:50.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:50 smithi122 ceph-mon[27145]: from='client.14624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:43:50.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:50 smithi122 ceph-mon[27145]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:43:51.291 INFO:teuthology.orchestra.run.smithi053.stdout: 5 35282 17.06 MB/sec warmup 61 sec latency 19.280 ms 2024-04-02T04:43:51.427 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:43:51.427 INFO:teuthology.orchestra.run.smithi053.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca", 2024-04-02T04:43:51.427 INFO:teuthology.orchestra.run.smithi053.stdout: "in_progress": true, 2024-04-02T04:43:51.427 INFO:teuthology.orchestra.run.smithi053.stdout: "services_complete": [], 2024-04-02T04:43:51.427 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": "", 2024-04-02T04:43:51.428 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:6cec793d9073f0594e2364b9b687ecaf22424cca image" 2024-04-02T04:43:51.428 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:43:51.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:51 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/1532279910' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:43:51.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:51 smithi053 ceph-mon[24889]: from='client.14636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:43:51.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:51 smithi053 ceph-mon[24889]: pgmap v267: 97 pgs: 97 active+clean; 227 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.3 MiB/s wr, 898 op/s 2024-04-02T04:43:51.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:51 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/trash_purge_schedule"}]: dispatch 2024-04-02T04:43:51.707 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:51 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:43:51.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:51 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/1532279910' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:43:51.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:51 smithi122 ceph-mon[27145]: from='client.14636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:43:51.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:51 smithi122 ceph-mon[27145]: pgmap v267: 97 pgs: 97 active+clean; 227 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.3 MiB/s wr, 898 op/s 2024-04-02T04:43:51.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:51 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/trash_purge_schedule"}]: dispatch 2024-04-02T04:43:51.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:51 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:43:52.056 INFO:teuthology.orchestra.run.smithi053.stdout:HEALTH_OK 2024-04-02T04:43:52.291 INFO:teuthology.orchestra.run.smithi053.stdout: 5 35861 17.10 MB/sec warmup 62 sec latency 24.132 ms 2024-04-02T04:43:52.713 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:52 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/1215093963' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:43:52.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:52 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/1215093963' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:43:53.291 INFO:teuthology.orchestra.run.smithi053.stdout: 5 36383 16.92 MB/sec warmup 63 sec latency 20.555 ms 2024-04-02T04:43:54.221 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:53 smithi053 ceph-mon[24889]: pgmap v268: 97 pgs: 97 active+clean; 233 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 1.09k op/s 2024-04-02T04:43:54.292 INFO:teuthology.orchestra.run.smithi053.stdout: 5 37205 16.99 MB/sec warmup 64 sec latency 21.584 ms 2024-04-02T04:43:54.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:53 smithi122 ceph-mon[27145]: pgmap v268: 97 pgs: 97 active+clean; 233 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 1.09k op/s 2024-04-02T04:43:55.292 INFO:teuthology.orchestra.run.smithi053.stdout: 5 37818 16.87 MB/sec warmup 65 sec latency 126.318 ms 2024-04-02T04:43:55.636 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:55 smithi053 ceph-mon[24889]: pgmap v269: 97 pgs: 97 active+clean; 236 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.3 MiB/s wr, 797 op/s 2024-04-02T04:43:55.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:55 smithi122 ceph-mon[27145]: pgmap v269: 97 pgs: 97 active+clean; 236 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.3 MiB/s wr, 797 op/s 2024-04-02T04:43:56.292 INFO:teuthology.orchestra.run.smithi053.stdout: 5 38653 17.10 MB/sec warmup 66 sec latency 75.015 ms 2024-04-02T04:43:57.292 INFO:teuthology.orchestra.run.smithi053.stdout: 5 39264 17.20 MB/sec warmup 67 sec latency 30.040 ms 2024-04-02T04:43:57.652 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:57 smithi053 ceph-mon[24889]: pgmap v270: 97 pgs: 97 active+clean; 240 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 884 op/s 2024-04-02T04:43:57.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:57 smithi122 ceph-mon[27145]: pgmap v270: 97 pgs: 97 active+clean; 240 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 884 op/s 2024-04-02T04:43:58.292 INFO:teuthology.orchestra.run.smithi053.stdout: 5 39750 17.05 MB/sec warmup 68 sec latency 30.877 ms 2024-04-02T04:43:59.292 INFO:teuthology.orchestra.run.smithi053.stdout: 5 40335 16.97 MB/sec warmup 69 sec latency 31.447 ms 2024-04-02T04:43:59.414 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:59 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:43:59.415 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:59 smithi053 ceph-mon[24889]: Upgrade: Target is version 19.0.0-1979-g6cec793d (unknown) 2024-04-02T04:43:59.415 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:59 smithi053 ceph-mon[24889]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae'] 2024-04-02T04:43:59.415 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:59 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:43:59.415 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:59 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:43:59.415 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:59 smithi053 ceph-mon[24889]: Upgrade: Need to upgrade myself (mgr.smithi053.legdmv) 2024-04-02T04:43:59.415 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:43:59 smithi053 ceph-mon[24889]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae on smithi122 2024-04-02T04:43:59.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:59 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:43:59.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:59 smithi122 ceph-mon[27145]: Upgrade: Target is version 19.0.0-1979-g6cec793d (unknown) 2024-04-02T04:43:59.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:59 smithi122 ceph-mon[27145]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae'] 2024-04-02T04:43:59.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:59 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:43:59.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:59 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:43:59.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:59 smithi122 ceph-mon[27145]: Upgrade: Need to upgrade myself (mgr.smithi053.legdmv) 2024-04-02T04:43:59.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:43:59 smithi122 ceph-mon[27145]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae on smithi122 2024-04-02T04:44:00.292 INFO:teuthology.orchestra.run.smithi053.stdout: 5 41138 16.97 MB/sec warmup 70 sec latency 16.810 ms 2024-04-02T04:44:00.418 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:00 smithi053 ceph-mon[24889]: pgmap v271: 97 pgs: 97 active+clean; 246 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.9 MiB/s wr, 1.02k op/s 2024-04-02T04:44:00.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:00 smithi122 ceph-mon[27145]: pgmap v271: 97 pgs: 97 active+clean; 246 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.9 MiB/s wr, 1.02k op/s 2024-04-02T04:44:01.297 INFO:teuthology.orchestra.run.smithi053.stdout: 5 41995 17.11 MB/sec warmup 71 sec latency 16.925 ms 2024-04-02T04:44:01.673 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:01 smithi053 ceph-mon[24889]: pgmap v272: 97 pgs: 97 active+clean; 247 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.6 MiB/s wr, 759 op/s 2024-04-02T04:44:01.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:01 smithi122 ceph-mon[27145]: pgmap v272: 97 pgs: 97 active+clean; 247 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.6 MiB/s wr, 759 op/s 2024-04-02T04:44:02.297 INFO:teuthology.orchestra.run.smithi053.stdout: 5 42990 17.37 MB/sec warmup 72 sec latency 15.645 ms 2024-04-02T04:44:03.298 INFO:teuthology.orchestra.run.smithi053.stdout: 5 43400 17.20 MB/sec warmup 73 sec latency 44.724 ms 2024-04-02T04:44:04.014 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:03 smithi053 ceph-mon[24889]: pgmap v273: 97 pgs: 97 active+clean; 257 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 1.05k op/s 2024-04-02T04:44:04.300 INFO:teuthology.orchestra.run.smithi053.stdout: 5 43880 17.11 MB/sec warmup 74 sec latency 34.863 ms 2024-04-02T04:44:04.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:03 smithi122 ceph-mon[27145]: pgmap v273: 97 pgs: 97 active+clean; 257 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 1.05k op/s 2024-04-02T04:44:05.300 INFO:teuthology.orchestra.run.smithi053.stdout: 5 44799 17.16 MB/sec warmup 75 sec latency 25.176 ms 2024-04-02T04:44:05.771 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:05 smithi053 ceph-mon[24889]: pgmap v274: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.7 MiB/s wr, 825 op/s 2024-04-02T04:44:05.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:05 smithi122 ceph-mon[27145]: pgmap v274: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.7 MiB/s wr, 825 op/s 2024-04-02T04:44:06.300 INFO:teuthology.orchestra.run.smithi053.stdout: 5 45685 17.33 MB/sec warmup 76 sec latency 22.461 ms 2024-04-02T04:44:07.300 INFO:teuthology.orchestra.run.smithi053.stdout: 5 46621 17.50 MB/sec warmup 77 sec latency 20.315 ms 2024-04-02T04:44:07.781 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:07 smithi053 ceph-mon[24889]: pgmap v275: 97 pgs: 97 active+clean; 267 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 11 MiB/s wr, 1.02k op/s 2024-04-02T04:44:07.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:07 smithi122 ceph-mon[27145]: pgmap v275: 97 pgs: 97 active+clean; 267 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 11 MiB/s wr, 1.02k op/s 2024-04-02T04:44:08.300 INFO:teuthology.orchestra.run.smithi053.stdout: 5 47320 17.44 MB/sec warmup 78 sec latency 24.128 ms 2024-04-02T04:44:09.300 INFO:teuthology.orchestra.run.smithi053.stdout: 5 47839 17.34 MB/sec warmup 79 sec latency 53.892 ms 2024-04-02T04:44:09.794 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:09 smithi053 ceph-mon[24889]: pgmap v276: 97 pgs: 97 active+clean; 269 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 1k op/s 2024-04-02T04:44:09.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:09 smithi122 ceph-mon[27145]: pgmap v276: 97 pgs: 97 active+clean; 269 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 1k op/s 2024-04-02T04:44:10.300 INFO:teuthology.orchestra.run.smithi053.stdout: 5 48442 17.31 MB/sec warmup 80 sec latency 32.413 ms 2024-04-02T04:44:11.300 INFO:teuthology.orchestra.run.smithi053.stdout: 5 48968 17.29 MB/sec warmup 81 sec latency 41.517 ms 2024-04-02T04:44:11.829 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:11 smithi053 ceph-mon[24889]: pgmap v277: 97 pgs: 97 active+clean; 269 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.1 MiB/s wr, 810 op/s 2024-04-02T04:44:11.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:11 smithi122 ceph-mon[27145]: pgmap v277: 97 pgs: 97 active+clean; 269 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.1 MiB/s wr, 810 op/s 2024-04-02T04:44:12.300 INFO:teuthology.orchestra.run.smithi053.stdout: 5 49808 17.47 MB/sec warmup 82 sec latency 21.599 ms 2024-04-02T04:44:13.300 INFO:teuthology.orchestra.run.smithi053.stdout: 5 50402 17.42 MB/sec warmup 83 sec latency 18.053 ms 2024-04-02T04:44:14.090 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:13 smithi053 ceph-mon[24889]: pgmap v278: 97 pgs: 97 active+clean; 275 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 1.05k op/s 2024-04-02T04:44:14.300 INFO:teuthology.orchestra.run.smithi053.stdout: 5 51187 17.42 MB/sec warmup 84 sec latency 23.394 ms 2024-04-02T04:44:14.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:13 smithi122 ceph-mon[27145]: pgmap v278: 97 pgs: 97 active+clean; 275 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 1.05k op/s 2024-04-02T04:44:15.300 INFO:teuthology.orchestra.run.smithi053.stdout: 5 52070 17.43 MB/sec warmup 85 sec latency 36.333 ms 2024-04-02T04:44:15.847 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:15 smithi053 ceph-mon[24889]: pgmap v279: 97 pgs: 97 active+clean; 282 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.9 MiB/s wr, 878 op/s 2024-04-02T04:44:15.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:15 smithi122 ceph-mon[27145]: pgmap v279: 97 pgs: 97 active+clean; 282 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.9 MiB/s wr, 878 op/s 2024-04-02T04:44:16.300 INFO:teuthology.orchestra.run.smithi053.stdout: 5 52779 17.54 MB/sec warmup 86 sec latency 22.646 ms 2024-04-02T04:44:17.301 INFO:teuthology.orchestra.run.smithi053.stdout: 5 53296 17.55 MB/sec warmup 87 sec latency 51.986 ms 2024-04-02T04:44:17.882 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:17 smithi053 ceph-mon[24889]: pgmap v280: 97 pgs: 97 active+clean; 282 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 882 op/s 2024-04-02T04:44:17.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:17 smithi122 ceph-mon[27145]: pgmap v280: 97 pgs: 97 active+clean; 282 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 882 op/s 2024-04-02T04:44:18.300 INFO:teuthology.orchestra.run.smithi053.stdout: 5 53853 17.51 MB/sec warmup 88 sec latency 29.628 ms 2024-04-02T04:44:19.300 INFO:teuthology.orchestra.run.smithi053.stdout: 5 54530 17.45 MB/sec warmup 89 sec latency 32.831 ms 2024-04-02T04:44:19.640 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:19 smithi053 ceph-mon[24889]: pgmap v281: 97 pgs: 97 active+clean; 291 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.2 MiB/s wr, 859 op/s 2024-04-02T04:44:19.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:19 smithi122 ceph-mon[27145]: pgmap v281: 97 pgs: 97 active+clean; 291 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.2 MiB/s wr, 859 op/s 2024-04-02T04:44:20.300 INFO:teuthology.orchestra.run.smithi053.stdout: 5 55306 17.44 MB/sec warmup 90 sec latency 25.458 ms 2024-04-02T04:44:21.300 INFO:teuthology.orchestra.run.smithi053.stdout: 5 56126 17.53 MB/sec warmup 91 sec latency 25.466 ms 2024-04-02T04:44:21.847 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:21 smithi053 ceph-mon[24889]: pgmap v282: 97 pgs: 97 active+clean; 291 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.4 MiB/s wr, 743 op/s 2024-04-02T04:44:21.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:21 smithi122 ceph-mon[27145]: pgmap v282: 97 pgs: 97 active+clean; 291 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.4 MiB/s wr, 743 op/s 2024-04-02T04:44:22.300 INFO:teuthology.orchestra.run.smithi053.stdout: 5 56719 17.61 MB/sec warmup 92 sec latency 58.495 ms 2024-04-02T04:44:22.485 INFO:teuthology.orchestra.run.smithi053.stdout:true 2024-04-02T04:44:22.860 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:22 smithi053 ceph-mon[24889]: from='client.14644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:44:22.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:22 smithi122 ceph-mon[27145]: from='client.14644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:44:23.300 INFO:teuthology.orchestra.run.smithi053.stdout: 5 57197 17.56 MB/sec warmup 93 sec latency 23.452 ms 2024-04-02T04:44:23.334 INFO:teuthology.orchestra.run.smithi053.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-02T04:44:23.334 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager.smithi053 smithi053 *:9093,9094 running (6m) 95s ago 7m 23.3M - ba2b418f427c 8d27e10ebf84 2024-04-02T04:44:23.334 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi053 smithi053 running (7m) 95s ago 7m 6409k - 17.2.0 e1d6a67b021e a5c436bbd36d 2024-04-02T04:44:23.334 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi122 smithi122 running (7m) 97s ago 7m 6405k - 17.2.0 e1d6a67b021e b7074474cd46 2024-04-02T04:44:23.334 INFO:teuthology.orchestra.run.smithi053.stdout:grafana.smithi053 smithi053 *:3000 running (6m) 95s ago 7m 52.0M - 8.3.5 dad864ee21e9 8f4a8d871d62 2024-04-02T04:44:23.334 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi053.nfrfot smithi053 running (2m) 95s ago 2m 14.9M - 17.2.0 e1d6a67b021e c94e9e643514 2024-04-02T04:44:23.334 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi122.mwoyoh smithi122 running (2m) 97s ago 2m 13.4M - 17.2.0 e1d6a67b021e 73376d9ee4b8 2024-04-02T04:44:23.334 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi053.legdmv smithi053 *:9283 running (9m) 95s ago 9m 460M - 17.2.0 e1d6a67b021e 5616241666e4 2024-04-02T04:44:23.334 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi122.cpwlyk smithi122 *:8443,9283 running (7m) 97s ago 7m 412M - 17.2.0 e1d6a67b021e b28b3080c55d 2024-04-02T04:44:23.335 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi053 smithi053 running (9m) 95s ago 9m 47.5M 2048M 17.2.0 e1d6a67b021e 665e4412f6da 2024-04-02T04:44:23.335 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi122 smithi122 running (7m) 97s ago 7m 36.2M 2048M 17.2.0 e1d6a67b021e a2df07e0883d 2024-04-02T04:44:23.335 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.0.0.smithi053.mhkdcb smithi053 *:2049 running (106s) 95s ago 105s 46.9M - 4.0 e1d6a67b021e 07e5da7bd2c5 2024-04-02T04:44:23.335 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.1.0.smithi122.mauvsb smithi122 *:2049 running (101s) 97s ago 100s 48.7M - 4.0 e1d6a67b021e 4f866685005e 2024-04-02T04:44:23.335 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi053 smithi053 *:9100 running (7m) 95s ago 7m 16.8M - 1dbe0e931976 87185e572b31 2024-04-02T04:44:23.335 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi122 smithi122 *:9100 running (7m) 97s ago 7m 19.2M - 1dbe0e931976 d576cf080067 2024-04-02T04:44:23.335 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 smithi053 running (6m) 95s ago 6m 42.8M 3217M 17.2.0 e1d6a67b021e 906cb1fd8042 2024-04-02T04:44:23.335 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 smithi053 running (5m) 95s ago 5m 45.9M 3217M 17.2.0 e1d6a67b021e ac1ad6266e63 2024-04-02T04:44:23.335 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 smithi053 running (5m) 95s ago 5m 40.3M 3217M 17.2.0 e1d6a67b021e 86964c8a94ff 2024-04-02T04:44:23.335 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 smithi053 running (5m) 95s ago 5m 39.8M 3217M 17.2.0 e1d6a67b021e 7e8b9e98ae2b 2024-04-02T04:44:23.335 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 smithi122 running (4m) 97s ago 4m 41.0M 3984M 17.2.0 e1d6a67b021e 46076afcd7bc 2024-04-02T04:44:23.335 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 smithi122 running (4m) 97s ago 4m 41.2M 3984M 17.2.0 e1d6a67b021e e687a2f6da4d 2024-04-02T04:44:23.335 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 smithi122 running (3m) 97s ago 3m 38.8M 3984M 17.2.0 e1d6a67b021e 9e371df9a839 2024-04-02T04:44:23.335 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 smithi122 running (3m) 97s ago 3m 40.3M 3984M 17.2.0 e1d6a67b021e 9b3cc398254d 2024-04-02T04:44:23.335 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus.smithi053 smithi053 *:9095 running (6m) 95s ago 6m 62.1M - 514e6a882f6e 36d493ce400e 2024-04-02T04:44:23.847 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:44:23.848 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": { 2024-04-02T04:44:23.848 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:44:23.848 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:44:23.848 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": { 2024-04-02T04:44:23.848 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:44:23.848 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:44:23.848 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": { 2024-04-02T04:44:23.848 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-02T04:44:23.848 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:44:23.848 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": { 2024-04-02T04:44:23.848 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:44:23.848 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:44:23.848 INFO:teuthology.orchestra.run.smithi053.stdout: "overall": { 2024-04-02T04:44:23.849 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-04-02T04:44:23.849 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-02T04:44:23.849 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:44:24.155 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:24 smithi053 ceph-mon[24889]: from='client.14648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:44:24.156 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:24 smithi053 ceph-mon[24889]: from='client.14652 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:44:24.156 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:24 smithi053 ceph-mon[24889]: pgmap v283: 97 pgs: 97 active+clean; 299 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 955 op/s 2024-04-02T04:44:24.156 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:24 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/265727976' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:44:24.300 INFO:teuthology.orchestra.run.smithi053.stdout: 5 57648 17.47 MB/sec warmup 94 sec latency 51.431 ms 2024-04-02T04:44:24.302 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:44:24.302 INFO:teuthology.orchestra.run.smithi053.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae", 2024-04-02T04:44:24.303 INFO:teuthology.orchestra.run.smithi053.stdout: "in_progress": true, 2024-04-02T04:44:24.303 INFO:teuthology.orchestra.run.smithi053.stdout: "services_complete": [], 2024-04-02T04:44:24.303 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": "0/23 daemons upgraded", 2024-04-02T04:44:24.303 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae image on host smithi122" 2024-04-02T04:44:24.303 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:44:24.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:24 smithi122 ceph-mon[27145]: from='client.14648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:44:24.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:24 smithi122 ceph-mon[27145]: from='client.14652 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:44:24.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:24 smithi122 ceph-mon[27145]: pgmap v283: 97 pgs: 97 active+clean; 299 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 955 op/s 2024-04-02T04:44:24.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:24 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/265727976' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:44:24.887 INFO:teuthology.orchestra.run.smithi053.stdout:HEALTH_OK 2024-04-02T04:44:25.177 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:25 smithi053 ceph-mon[24889]: from='client.24449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:44:25.177 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:25 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/2114922912' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:44:25.301 INFO:teuthology.orchestra.run.smithi053.stdout: 5 58244 17.40 MB/sec warmup 95 sec latency 26.199 ms 2024-04-02T04:44:25.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:25 smithi122 ceph-mon[27145]: from='client.24449 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:44:25.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:25 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/2114922912' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:44:26.194 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:26 smithi053 ceph-mon[24889]: pgmap v284: 97 pgs: 97 active+clean; 304 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.7 MiB/s wr, 872 op/s 2024-04-02T04:44:26.300 INFO:teuthology.orchestra.run.smithi053.stdout: 5 58828 17.35 MB/sec warmup 96 sec latency 31.325 ms 2024-04-02T04:44:26.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:26 smithi122 ceph-mon[27145]: pgmap v284: 97 pgs: 97 active+clean; 304 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.7 MiB/s wr, 872 op/s 2024-04-02T04:44:27.302 INFO:teuthology.orchestra.run.smithi053.stdout: 5 59324 17.30 MB/sec warmup 97 sec latency 49.406 ms 2024-04-02T04:44:27.718 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:27 smithi053 ceph-mon[24889]: pgmap v285: 97 pgs: 97 active+clean; 302 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.4 MiB/s wr, 875 op/s 2024-04-02T04:44:27.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:27 smithi122 ceph-mon[27145]: pgmap v285: 97 pgs: 97 active+clean; 302 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.4 MiB/s wr, 875 op/s 2024-04-02T04:44:28.301 INFO:teuthology.orchestra.run.smithi053.stdout: 5 59853 17.33 MB/sec warmup 98 sec latency 42.437 ms 2024-04-02T04:44:29.301 INFO:teuthology.orchestra.run.smithi053.stdout: 5 60511 17.40 MB/sec warmup 99 sec latency 27.247 ms 2024-04-02T04:44:29.743 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:29 smithi053 ceph-mon[24889]: pgmap v286: 97 pgs: 97 active+clean; 304 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 8.7 MiB/s wr, 904 op/s 2024-04-02T04:44:29.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:29 smithi122 ceph-mon[27145]: pgmap v286: 97 pgs: 97 active+clean; 304 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 8.7 MiB/s wr, 904 op/s 2024-04-02T04:44:30.301 INFO:teuthology.orchestra.run.smithi053.stdout: 5 60935 17.32 MB/sec warmup 100 sec latency 23.367 ms 2024-04-02T04:44:31.301 INFO:teuthology.orchestra.run.smithi053.stdout: 5 61374 17.22 MB/sec warmup 101 sec latency 46.609 ms 2024-04-02T04:44:31.761 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:31 smithi053 ceph-mon[24889]: pgmap v287: 97 pgs: 97 active+clean; 304 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 7.5 MiB/s wr, 687 op/s 2024-04-02T04:44:31.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:31 smithi122 ceph-mon[27145]: pgmap v287: 97 pgs: 97 active+clean; 304 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 7.5 MiB/s wr, 687 op/s 2024-04-02T04:44:32.301 INFO:teuthology.orchestra.run.smithi053.stdout: 5 61875 17.15 MB/sec warmup 102 sec latency 53.175 ms 2024-04-02T04:44:33.301 INFO:teuthology.orchestra.run.smithi053.stdout: 5 62394 17.10 MB/sec warmup 103 sec latency 26.890 ms 2024-04-02T04:44:33.542 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:33 smithi053 ceph-mon[24889]: Upgrade: Updating mgr.smithi122.cpwlyk 2024-04-02T04:44:33.542 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:33 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:44:33.542 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:33 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi122.cpwlyk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-02T04:44:33.542 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:33 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-02T04:44:33.543 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:33 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:44:33.543 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:33 smithi053 ceph-mon[24889]: Deploying daemon mgr.smithi122.cpwlyk on smithi122 2024-04-02T04:44:33.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:33 smithi122 ceph-mon[27145]: Upgrade: Updating mgr.smithi122.cpwlyk 2024-04-02T04:44:33.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:33 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:44:33.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:33 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi122.cpwlyk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-02T04:44:33.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:33 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-02T04:44:33.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:33 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:44:33.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:33 smithi122 ceph-mon[27145]: Deploying daemon mgr.smithi122.cpwlyk on smithi122 2024-04-02T04:44:34.301 INFO:teuthology.orchestra.run.smithi053.stdout: 5 63031 17.10 MB/sec warmup 104 sec latency 30.088 ms 2024-04-02T04:44:34.545 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:34 smithi053 ceph-mon[24889]: pgmap v288: 97 pgs: 97 active+clean; 312 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.0 MiB/s wr, 835 op/s 2024-04-02T04:44:34.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:34 smithi122 ceph-mon[27145]: pgmap v288: 97 pgs: 97 active+clean; 312 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.0 MiB/s wr, 835 op/s 2024-04-02T04:44:35.301 INFO:teuthology.orchestra.run.smithi053.stdout: 5 63665 17.19 MB/sec warmup 105 sec latency 22.325 ms 2024-04-02T04:44:35.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:35 smithi122 ceph-mon[27145]: pgmap v289: 97 pgs: 97 active+clean; 318 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 7.9 MiB/s wr, 816 op/s 2024-04-02T04:44:35.812 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:35 smithi053 ceph-mon[24889]: pgmap v289: 97 pgs: 97 active+clean; 318 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 7.9 MiB/s wr, 816 op/s 2024-04-02T04:44:36.301 INFO:teuthology.orchestra.run.smithi053.stdout: 5 64179 17.19 MB/sec warmup 106 sec latency 26.798 ms 2024-04-02T04:44:37.301 INFO:teuthology.orchestra.run.smithi053.stdout: 5 64489 17.07 MB/sec warmup 107 sec latency 164.849 ms 2024-04-02T04:44:37.793 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:37 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:44:37.793 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:37 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:44:37.793 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:37 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:44:37.793 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:37 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:44:37.793 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:37 smithi122 ceph-mon[27145]: pgmap v290: 97 pgs: 97 active+clean; 319 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 7.6 MiB/s wr, 725 op/s 2024-04-02T04:44:37.828 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:37 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:44:37.828 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:37 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:44:37.829 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:37 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:44:37.829 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:37 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:44:37.829 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:37 smithi053 ceph-mon[24889]: pgmap v290: 97 pgs: 97 active+clean; 319 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 7.6 MiB/s wr, 725 op/s 2024-04-02T04:44:38.301 INFO:teuthology.orchestra.run.smithi053.stdout: 5 64876 16.99 MB/sec warmup 108 sec latency 36.852 ms 2024-04-02T04:44:39.301 INFO:teuthology.orchestra.run.smithi053.stdout: 5 65461 16.93 MB/sec warmup 109 sec latency 41.086 ms 2024-04-02T04:44:39.858 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:39 smithi053 ceph-mon[24889]: pgmap v291: 97 pgs: 97 active+clean; 328 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 7.8 MiB/s wr, 823 op/s 2024-04-02T04:44:39.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:39 smithi122 ceph-mon[27145]: pgmap v291: 97 pgs: 97 active+clean; 328 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 7.8 MiB/s wr, 823 op/s 2024-04-02T04:44:40.301 INFO:teuthology.orchestra.run.smithi053.stdout: 5 66077 16.90 MB/sec warmup 110 sec latency 29.382 ms 2024-04-02T04:44:41.301 INFO:teuthology.orchestra.run.smithi053.stdout: 5 66632 16.92 MB/sec warmup 111 sec latency 48.714 ms 2024-04-02T04:44:42.302 INFO:teuthology.orchestra.run.smithi053.stdout: 5 67234 16.99 MB/sec warmup 112 sec latency 24.902 ms 2024-04-02T04:44:42.523 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:42 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:44:42.523 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:42 smithi053 ceph-mon[24889]: pgmap v292: 97 pgs: 97 active+clean; 326 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 7.1 MiB/s wr, 737 op/s 2024-04-02T04:44:42.523 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:42 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:44:42.806 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:42 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:44:42.807 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:42 smithi122 ceph-mon[27145]: pgmap v292: 97 pgs: 97 active+clean; 326 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 7.1 MiB/s wr, 737 op/s 2024-04-02T04:44:42.807 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:42 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:44:43.301 INFO:teuthology.orchestra.run.smithi053.stdout: 5 67639 16.95 MB/sec warmup 113 sec latency 27.234 ms 2024-04-02T04:44:44.047 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:44 smithi053 ceph-mon[24889]: pgmap v293: 97 pgs: 97 active+clean; 329 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8.8 MiB/s wr, 960 op/s 2024-04-02T04:44:44.237 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:44 smithi122 ceph-mon[27145]: pgmap v293: 97 pgs: 97 active+clean; 329 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8.8 MiB/s wr, 960 op/s 2024-04-02T04:44:44.302 INFO:teuthology.orchestra.run.smithi053.stdout: 5 67997 16.86 MB/sec warmup 114 sec latency 39.651 ms 2024-04-02T04:44:45.302 INFO:teuthology.orchestra.run.smithi053.stdout: 5 68299 16.75 MB/sec warmup 115 sec latency 32.020 ms 2024-04-02T04:44:45.566 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:45 smithi053 ceph-mon[24889]: Standby manager daemon smithi122.cpwlyk restarted 2024-04-02T04:44:45.566 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:45 smithi053 ceph-mon[24889]: Standby manager daemon smithi122.cpwlyk started 2024-04-02T04:44:45.566 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:45 smithi053 ceph-mon[24889]: from='mgr.? 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.cpwlyk/crt"}]: dispatch 2024-04-02T04:44:45.567 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:45 smithi053 ceph-mon[24889]: from='mgr.? 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-02T04:44:45.567 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:45 smithi053 ceph-mon[24889]: from='mgr.? 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.cpwlyk/key"}]: dispatch 2024-04-02T04:44:45.567 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:45 smithi053 ceph-mon[24889]: from='mgr.? 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-02T04:44:45.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:45 smithi122 ceph-mon[27145]: Standby manager daemon smithi122.cpwlyk restarted 2024-04-02T04:44:45.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:45 smithi122 ceph-mon[27145]: Standby manager daemon smithi122.cpwlyk started 2024-04-02T04:44:45.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:45 smithi122 ceph-mon[27145]: from='mgr.? 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.cpwlyk/crt"}]: dispatch 2024-04-02T04:44:45.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:45 smithi122 ceph-mon[27145]: from='mgr.? 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-02T04:44:45.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:45 smithi122 ceph-mon[27145]: from='mgr.? 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.cpwlyk/key"}]: dispatch 2024-04-02T04:44:45.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:45 smithi122 ceph-mon[27145]: from='mgr.? 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-02T04:44:46.302 INFO:teuthology.orchestra.run.smithi053.stdout: 5 68806 16.72 MB/sec warmup 116 sec latency 43.837 ms 2024-04-02T04:44:46.582 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:46 smithi053 ceph-mon[24889]: mgrmap e22: smithi053.legdmv(active, since 7m), standbys: smithi122.cpwlyk 2024-04-02T04:44:46.583 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:46 smithi053 ceph-mon[24889]: pgmap v294: 97 pgs: 97 active+clean; 333 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 7.8 MiB/s wr, 838 op/s 2024-04-02T04:44:46.583 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:46 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:44:46.583 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:46 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:44:46.583 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:46 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:44:46.583 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:46 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:44:46.583 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:46 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mgr fail", "who": "smithi053.legdmv"}]: dispatch 2024-04-02T04:44:46.583 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:46 smithi053 ceph-mon[24889]: osdmap e70: 8 total, 8 up, 8 in 2024-04-02T04:44:46.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:46 smithi122 ceph-mon[27145]: mgrmap e22: smithi053.legdmv(active, since 7m), standbys: smithi122.cpwlyk 2024-04-02T04:44:46.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:46 smithi122 ceph-mon[27145]: pgmap v294: 97 pgs: 97 active+clean; 333 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 7.8 MiB/s wr, 838 op/s 2024-04-02T04:44:46.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:46 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:44:46.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:46 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:44:46.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:46 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' 2024-04-02T04:44:46.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:46 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:44:46.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:46 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mgr fail", "who": "smithi053.legdmv"}]: dispatch 2024-04-02T04:44:46.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:46 smithi122 ceph-mon[27145]: osdmap e70: 8 total, 8 up, 8 in 2024-04-02T04:44:47.304 INFO:teuthology.orchestra.run.smithi053.stdout: 5 69358 16.67 MB/sec warmup 117 sec latency 46.883 ms 2024-04-02T04:44:47.629 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:47 smithi053 ceph-mon[24889]: Upgrade: Need to upgrade myself (mgr.smithi053.legdmv) 2024-04-02T04:44:47.629 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:47 smithi053 ceph-mon[24889]: Failing over to other MGR 2024-04-02T04:44:47.629 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:47 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "mon metadata", "id": "smithi053"}]: dispatch 2024-04-02T04:44:47.629 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:47 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-02T04:44:47.629 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:47 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi053.nfrfot"}]: dispatch 2024-04-02T04:44:47.629 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:47 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi122.mwoyoh"}]: dispatch 2024-04-02T04:44:47.629 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:47 smithi053 ceph-mon[24889]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "mgr fail", "who": "smithi053.legdmv"}]': finished 2024-04-02T04:44:47.630 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:47 smithi053 ceph-mon[24889]: mgrmap e23: smithi122.cpwlyk(active, starting, since 1.00416s) 2024-04-02T04:44:47.630 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:47 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "mgr metadata", "who": "smithi122.cpwlyk", "id": "smithi122.cpwlyk"}]: dispatch 2024-04-02T04:44:47.630 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:47 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:44:47.630 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:47 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:44:47.630 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:47 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:44:47.630 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:47 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:44:47.630 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:47 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:44:47.630 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:47 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:44:47.630 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:47 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:44:47.630 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:47 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:44:47.630 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:47 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-02T04:44:47.630 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:47 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-02T04:44:47.630 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:47 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-02T04:44:47.630 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:47 smithi053 ceph-mon[24889]: Manager daemon smithi122.cpwlyk is now available 2024-04-02T04:44:47.631 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:47 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:44:47.631 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:47 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:44:47.631 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:47 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:44:47.631 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:47 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi122.cpwlyk/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:44:47.631 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:47 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi122.cpwlyk/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:44:47.631 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:47 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi122.cpwlyk/trash_purge_schedule"}]: dispatch 2024-04-02T04:44:47.631 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:47 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi122.cpwlyk/trash_purge_schedule"}]: dispatch 2024-04-02T04:44:47.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:47 smithi122 ceph-mon[27145]: Upgrade: Need to upgrade myself (mgr.smithi053.legdmv) 2024-04-02T04:44:47.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:47 smithi122 ceph-mon[27145]: Failing over to other MGR 2024-04-02T04:44:47.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:47 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "mon metadata", "id": "smithi053"}]: dispatch 2024-04-02T04:44:47.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:47 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-02T04:44:47.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:47 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi053.nfrfot"}]: dispatch 2024-04-02T04:44:47.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:47 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi122.mwoyoh"}]: dispatch 2024-04-02T04:44:47.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:47 smithi122 ceph-mon[27145]: from='mgr.14206 172.21.15.53:0/4236504339' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "mgr fail", "who": "smithi053.legdmv"}]': finished 2024-04-02T04:44:47.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:47 smithi122 ceph-mon[27145]: mgrmap e23: smithi122.cpwlyk(active, starting, since 1.00416s) 2024-04-02T04:44:47.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:47 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "mgr metadata", "who": "smithi122.cpwlyk", "id": "smithi122.cpwlyk"}]: dispatch 2024-04-02T04:44:47.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:47 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:44:47.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:47 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:44:47.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:47 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:44:47.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:47 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:44:47.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:47 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:44:47.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:47 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:44:47.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:47 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:44:47.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:47 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:44:47.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:47 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-02T04:44:47.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:47 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-02T04:44:47.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:47 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-02T04:44:47.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:47 smithi122 ceph-mon[27145]: Manager daemon smithi122.cpwlyk is now available 2024-04-02T04:44:47.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:47 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:44:47.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:47 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:44:47.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:47 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:44:47.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:47 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi122.cpwlyk/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:44:47.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:47 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi122.cpwlyk/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:44:47.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:47 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi122.cpwlyk/trash_purge_schedule"}]: dispatch 2024-04-02T04:44:47.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:47 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi122.cpwlyk/trash_purge_schedule"}]: dispatch 2024-04-02T04:44:48.304 INFO:teuthology.orchestra.run.smithi053.stdout: 5 69919 16.65 MB/sec warmup 118 sec latency 33.448 ms 2024-04-02T04:44:48.639 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:48 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:44:48.639 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:48 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:44:48.639 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:48 smithi053 ceph-mon[24889]: mgrmap e24: smithi122.cpwlyk(active, since 2s) 2024-04-02T04:44:48.705 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:48 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:44:48.706 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:48 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:44:48.706 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:48 smithi122 ceph-mon[27145]: mgrmap e24: smithi122.cpwlyk(active, since 2s) 2024-04-02T04:44:49.304 INFO:teuthology.orchestra.run.smithi053.stdout: 5 70592 16.71 MB/sec warmup 119 sec latency 21.514 ms 2024-04-02T04:44:49.648 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:49 smithi053 ceph-mon[24889]: Deploying cephadm binary to smithi122 2024-04-02T04:44:49.648 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:49 smithi053 ceph-mon[24889]: pgmap v3: 97 pgs: 97 active+clean; 321 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-04-02T04:44:49.649 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:49 smithi053 ceph-mon[24889]: [02/Apr/2024:04:44:48] ENGINE Bus STARTING 2024-04-02T04:44:49.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:49 smithi122 ceph-mon[27145]: Deploying cephadm binary to smithi122 2024-04-02T04:44:49.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:49 smithi122 ceph-mon[27145]: pgmap v3: 97 pgs: 97 active+clean; 321 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-04-02T04:44:49.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:49 smithi122 ceph-mon[27145]: [02/Apr/2024:04:44:48] ENGINE Bus STARTING 2024-04-02T04:44:50.401 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:50 smithi053 ceph-mon[24889]: [02/Apr/2024:04:44:48] ENGINE Serving on http://172.21.15.122:8765 2024-04-02T04:44:50.652 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:50 smithi053 ceph-mon[24889]: [02/Apr/2024:04:44:48] ENGINE Serving on https://172.21.15.122:7150 2024-04-02T04:44:50.653 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:50 smithi053 ceph-mon[24889]: [02/Apr/2024:04:44:48] ENGINE Bus STARTED 2024-04-02T04:44:50.653 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:50 smithi053 ceph-mon[24889]: pgmap v4: 97 pgs: 97 active+clean; 321 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-04-02T04:44:50.653 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:50 smithi053 ceph-mon[24889]: mgrmap e25: smithi122.cpwlyk(active, since 4s) 2024-04-02T04:44:50.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:50 smithi122 ceph-mon[27145]: [02/Apr/2024:04:44:48] ENGINE Serving on http://172.21.15.122:8765 2024-04-02T04:44:50.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:50 smithi122 ceph-mon[27145]: [02/Apr/2024:04:44:48] ENGINE Serving on https://172.21.15.122:7150 2024-04-02T04:44:50.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:50 smithi122 ceph-mon[27145]: [02/Apr/2024:04:44:48] ENGINE Bus STARTED 2024-04-02T04:44:50.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:50 smithi122 ceph-mon[27145]: pgmap v4: 97 pgs: 97 active+clean; 321 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-04-02T04:44:50.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:50 smithi122 ceph-mon[27145]: mgrmap e25: smithi122.cpwlyk(active, since 4s) 2024-04-02T04:44:51.304 INFO:teuthology.orchestra.run.smithi053.stdout: 5 71484 10.86 MB/sec execute 1 sec latency 41.012 ms 2024-04-02T04:44:51.405 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:51 smithi053 ceph-mon[24889]: Deploying cephadm binary to smithi053 2024-04-02T04:44:51.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:51 smithi122 ceph-mon[27145]: Deploying cephadm binary to smithi053 2024-04-02T04:44:52.304 INFO:teuthology.orchestra.run.smithi053.stdout: 5 71918 9.61 MB/sec execute 2 sec latency 27.590 ms 2024-04-02T04:44:52.409 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:52 smithi053 ceph-mon[24889]: pgmap v5: 97 pgs: 97 active+clean; 321 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-04-02T04:44:52.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:52 smithi122 ceph-mon[27145]: pgmap v5: 97 pgs: 97 active+clean; 321 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-04-02T04:44:53.304 INFO:teuthology.orchestra.run.smithi053.stdout: 5 72374 9.74 MB/sec execute 3 sec latency 59.121 ms 2024-04-02T04:44:54.304 INFO:teuthology.orchestra.run.smithi053.stdout: 5 72786 9.06 MB/sec execute 4 sec latency 131.304 ms 2024-04-02T04:44:54.525 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:54 smithi053 ceph-mon[24889]: pgmap v6: 97 pgs: 97 active+clean; 321 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-04-02T04:44:54.711 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:54 smithi122 ceph-mon[27145]: pgmap v6: 97 pgs: 97 active+clean; 321 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-04-02T04:44:55.304 INFO:teuthology.orchestra.run.smithi053.stdout: 5 73338 9.81 MB/sec execute 5 sec latency 34.338 ms 2024-04-02T04:44:55.523 INFO:teuthology.orchestra.run.smithi053.stdout:true 2024-04-02T04:44:56.304 INFO:teuthology.orchestra.run.smithi053.stdout: 5 73861 11.02 MB/sec execute 6 sec latency 44.449 ms 2024-04-02T04:44:56.439 INFO:teuthology.orchestra.run.smithi053.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-02T04:44:56.440 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager.smithi053 smithi053 *:9093,9094 running (7m) 2m ago 8m 23.3M - ba2b418f427c 8d27e10ebf84 2024-04-02T04:44:56.440 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi053 smithi053 running (8m) 2m ago 8m 6409k - 17.2.0 e1d6a67b021e a5c436bbd36d 2024-04-02T04:44:56.440 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi122 smithi122 running (7m) 14s ago 7m 6405k - 17.2.0 e1d6a67b021e b7074474cd46 2024-04-02T04:44:56.440 INFO:teuthology.orchestra.run.smithi053.stdout:grafana.smithi053 smithi053 *:3000 running (6m) 2m ago 8m 52.0M - 8.3.5 dad864ee21e9 8f4a8d871d62 2024-04-02T04:44:56.440 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi053.nfrfot smithi053 running (2m) 2m ago 2m 14.9M - 17.2.0 e1d6a67b021e c94e9e643514 2024-04-02T04:44:56.440 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi122.mwoyoh smithi122 running (2m) 14s ago 2m 13.8M - 17.2.0 e1d6a67b021e 73376d9ee4b8 2024-04-02T04:44:56.440 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi053.legdmv smithi053 *:9283 running (9m) 2m ago 9m 460M - 17.2.0 e1d6a67b021e 5616241666e4 2024-04-02T04:44:56.440 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi122.cpwlyk smithi122 *:8443,9283 running (20s) 14s ago 7m 50.8M - 19.0.0-1979-g6cec793d 78f830190839 3a534988cf3b 2024-04-02T04:44:56.440 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi053 smithi053 running (9m) 2m ago 9m 47.5M 2048M 17.2.0 e1d6a67b021e 665e4412f6da 2024-04-02T04:44:56.440 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi122 smithi122 running (7m) 14s ago 7m 37.2M 2048M 17.2.0 e1d6a67b021e a2df07e0883d 2024-04-02T04:44:56.440 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.0.0.smithi053.mhkdcb smithi053 *:2049 running (2m) 2m ago 2m 46.9M - 4.0 e1d6a67b021e 07e5da7bd2c5 2024-04-02T04:44:56.440 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.1.0.smithi122.mauvsb smithi122 *:2049 running (2m) 14s ago 2m 73.2M - 4.0 e1d6a67b021e 4f866685005e 2024-04-02T04:44:56.441 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi053 smithi053 *:9100 running (8m) 2m ago 8m 16.8M - 1dbe0e931976 87185e572b31 2024-04-02T04:44:56.441 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi122 smithi122 *:9100 running (7m) 14s ago 7m 19.0M - 1dbe0e931976 d576cf080067 2024-04-02T04:44:56.441 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 smithi053 running (7m) 2m ago 7m 42.8M 3217M 17.2.0 e1d6a67b021e 906cb1fd8042 2024-04-02T04:44:56.441 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 smithi053 running (6m) 2m ago 6m 45.9M 3217M 17.2.0 e1d6a67b021e ac1ad6266e63 2024-04-02T04:44:56.441 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 smithi053 running (6m) 2m ago 6m 40.3M 3217M 17.2.0 e1d6a67b021e 86964c8a94ff 2024-04-02T04:44:56.441 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 smithi053 running (5m) 2m ago 5m 39.8M 3217M 17.2.0 e1d6a67b021e 7e8b9e98ae2b 2024-04-02T04:44:56.441 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 smithi122 running (5m) 14s ago 5m 243M 3984M 17.2.0 e1d6a67b021e 46076afcd7bc 2024-04-02T04:44:56.441 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 smithi122 running (4m) 14s ago 4m 269M 3984M 17.2.0 e1d6a67b021e e687a2f6da4d 2024-04-02T04:44:56.441 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 smithi122 running (4m) 14s ago 4m 245M 3984M 17.2.0 e1d6a67b021e 9e371df9a839 2024-04-02T04:44:56.441 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 smithi122 running (4m) 14s ago 4m 324M 3984M 17.2.0 e1d6a67b021e 9b3cc398254d 2024-04-02T04:44:56.441 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus.smithi053 smithi053 *:9095 running (7m) 2m ago 7m 62.1M - 514e6a882f6e 36d493ce400e 2024-04-02T04:44:56.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:56 smithi053 ceph-mon[24889]: pgmap v7: 97 pgs: 97 active+clean; 324 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 37 KiB/s rd, 6.0 MiB/s wr, 542 op/s 2024-04-02T04:44:56.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:56 smithi053 ceph-mon[24889]: Standby manager daemon smithi053.legdmv started 2024-04-02T04:44:56.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:56 smithi053 ceph-mon[24889]: from='mgr.? 172.21.15.53:0/57167772' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi053.legdmv/crt"}]: dispatch 2024-04-02T04:44:56.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:56 smithi053 ceph-mon[24889]: from='mgr.? 172.21.15.53:0/57167772' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-02T04:44:56.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:56 smithi053 ceph-mon[24889]: from='mgr.? 172.21.15.53:0/57167772' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi053.legdmv/key"}]: dispatch 2024-04-02T04:44:56.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:56 smithi053 ceph-mon[24889]: from='mgr.? 172.21.15.53:0/57167772' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-02T04:44:56.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:56 smithi122 ceph-mon[27145]: pgmap v7: 97 pgs: 97 active+clean; 324 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 37 KiB/s rd, 6.0 MiB/s wr, 542 op/s 2024-04-02T04:44:56.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:56 smithi122 ceph-mon[27145]: Standby manager daemon smithi053.legdmv started 2024-04-02T04:44:56.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:56 smithi122 ceph-mon[27145]: from='mgr.? 172.21.15.53:0/57167772' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi053.legdmv/crt"}]: dispatch 2024-04-02T04:44:56.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:56 smithi122 ceph-mon[27145]: from='mgr.? 172.21.15.53:0/57167772' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-02T04:44:56.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:56 smithi122 ceph-mon[27145]: from='mgr.? 172.21.15.53:0/57167772' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi053.legdmv/key"}]: dispatch 2024-04-02T04:44:56.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:56 smithi122 ceph-mon[27145]: from='mgr.? 172.21.15.53:0/57167772' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-02T04:44:56.970 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:44:56.970 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": { 2024-04-02T04:44:56.970 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:44:56.970 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:44:56.970 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": { 2024-04-02T04:44:56.970 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-02T04:44:56.970 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 1 2024-04-02T04:44:56.970 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:44:56.970 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": { 2024-04-02T04:44:56.970 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-02T04:44:56.971 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:44:56.971 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": { 2024-04-02T04:44:56.971 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:44:56.971 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:44:56.971 INFO:teuthology.orchestra.run.smithi053.stdout: "overall": { 2024-04-02T04:44:56.971 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-04-02T04:44:56.971 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 1 2024-04-02T04:44:56.971 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-02T04:44:56.971 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:44:57.304 INFO:teuthology.orchestra.run.smithi053.stdout: 5 74261 11.80 MB/sec execute 7 sec latency 109.513 ms 2024-04-02T04:44:57.515 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:44:57.515 INFO:teuthology.orchestra.run.smithi053.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae", 2024-04-02T04:44:57.515 INFO:teuthology.orchestra.run.smithi053.stdout: "in_progress": true, 2024-04-02T04:44:57.515 INFO:teuthology.orchestra.run.smithi053.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-02T04:44:57.515 INFO:teuthology.orchestra.run.smithi053.stdout: "services_complete": [], 2024-04-02T04:44:57.516 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": "1/23 daemons upgraded", 2024-04-02T04:44:57.516 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "", 2024-04-02T04:44:57.516 INFO:teuthology.orchestra.run.smithi053.stdout: "is_paused": false 2024-04-02T04:44:57.516 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:44:57.574 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:57 smithi053 ceph-mon[24889]: from='client.14690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:44:57.574 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:57 smithi053 ceph-mon[24889]: from='client.14694 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:44:57.575 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:57 smithi053 ceph-mon[24889]: mgrmap e26: smithi122.cpwlyk(active, since 10s), standbys: smithi053.legdmv 2024-04-02T04:44:57.575 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:57 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "mgr metadata", "who": "smithi053.legdmv", "id": "smithi053.legdmv"}]: dispatch 2024-04-02T04:44:57.575 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:57 smithi053 ceph-mon[24889]: from='client.24497 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:44:57.575 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:57 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/2074720192' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:44:57.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:57 smithi122 ceph-mon[27145]: from='client.14690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:44:57.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:57 smithi122 ceph-mon[27145]: from='client.14694 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:44:57.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:57 smithi122 ceph-mon[27145]: mgrmap e26: smithi122.cpwlyk(active, since 10s), standbys: smithi053.legdmv 2024-04-02T04:44:57.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:57 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "mgr metadata", "who": "smithi053.legdmv", "id": "smithi053.legdmv"}]: dispatch 2024-04-02T04:44:57.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:57 smithi122 ceph-mon[27145]: from='client.24497 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:44:57.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:57 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/2074720192' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:44:58.231 INFO:teuthology.orchestra.run.smithi053.stdout:HEALTH_OK 2024-04-02T04:44:58.304 INFO:teuthology.orchestra.run.smithi053.stdout: 5 74827 12.56 MB/sec execute 8 sec latency 20.000 ms 2024-04-02T04:44:58.578 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:58 smithi053 ceph-mon[24889]: pgmap v8: 97 pgs: 97 active+clean; 324 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 4.6 MiB/s wr, 421 op/s 2024-04-02T04:44:58.578 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:58 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/232953398' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:44:58.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:58 smithi122 ceph-mon[27145]: pgmap v8: 97 pgs: 97 active+clean; 324 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 4.6 MiB/s wr, 421 op/s 2024-04-02T04:44:58.599 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:58 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/232953398' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:44:59.304 INFO:teuthology.orchestra.run.smithi053.stdout: 5 75296 12.21 MB/sec execute 9 sec latency 28.176 ms 2024-04-02T04:44:59.583 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:44:59 smithi053 ceph-mon[24889]: from='client.14704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:44:59.729 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:44:59 smithi122 ceph-mon[27145]: from='client.14704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:45:00.304 INFO:teuthology.orchestra.run.smithi053.stdout: 5 75700 11.99 MB/sec execute 10 sec latency 61.505 ms 2024-04-02T04:45:00.592 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:00 smithi053 ceph-mon[24889]: pgmap v9: 97 pgs: 97 active+clean; 329 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 7.1 MiB/s wr, 643 op/s 2024-04-02T04:45:00.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:00 smithi122 ceph-mon[27145]: pgmap v9: 97 pgs: 97 active+clean; 329 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 7.1 MiB/s wr, 643 op/s 2024-04-02T04:45:01.304 INFO:teuthology.orchestra.run.smithi053.stdout: 5 76221 11.85 MB/sec execute 11 sec latency 39.087 ms 2024-04-02T04:45:01.877 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:01 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:01.877 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:01 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:01.877 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:01 smithi053 ceph-mon[24889]: pgmap v10: 97 pgs: 97 active+clean; 329 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 6.5 MiB/s wr, 586 op/s 2024-04-02T04:45:01.878 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:01 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:01.878 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:01 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:02.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:01 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:02.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:01 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:02.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:01 smithi122 ceph-mon[27145]: pgmap v10: 97 pgs: 97 active+clean; 329 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 6.5 MiB/s wr, 586 op/s 2024-04-02T04:45:02.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:01 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:02.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:01 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:02.304 INFO:teuthology.orchestra.run.smithi053.stdout: 5 76813 11.96 MB/sec execute 12 sec latency 32.329 ms 2024-04-02T04:45:02.916 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:02 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:45:02.916 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:02 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:02.916 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:02 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:03.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:02 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:45:03.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:02 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:03.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:02 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:03.304 INFO:teuthology.orchestra.run.smithi053.stdout: 5 77296 12.10 MB/sec execute 13 sec latency 41.780 ms 2024-04-02T04:45:04.174 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:03 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:04.174 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:03 smithi053 ceph-mon[24889]: pgmap v11: 97 pgs: 97 active+clean; 329 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 6.5 MiB/s wr, 586 op/s 2024-04-02T04:45:04.174 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:03 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:04.174 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:03 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:04.174 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:03 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:04.174 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:03 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:04.174 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:03 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:04.174 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:03 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:04.174 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:03 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:04.174 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:03 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:04.175 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:03 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:04.175 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:03 smithi053 ceph-mon[24889]: Adjusting osd_memory_target on smithi053 to 1937M 2024-04-02T04:45:04.175 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:03 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:04.212 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:03 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:04.212 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:03 smithi122 ceph-mon[27145]: pgmap v11: 97 pgs: 97 active+clean; 329 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 6.5 MiB/s wr, 586 op/s 2024-04-02T04:45:04.212 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:03 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:04.212 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:03 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:04.212 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:03 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:04.212 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:03 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:04.213 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:03 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:04.213 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:03 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:04.213 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:03 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:04.213 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:03 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:04.213 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:03 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:04.213 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:03 smithi122 ceph-mon[27145]: Adjusting osd_memory_target on smithi053 to 1937M 2024-04-02T04:45:04.213 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:03 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:04.304 INFO:teuthology.orchestra.run.smithi053.stdout: 5 77835 12.76 MB/sec execute 14 sec latency 25.919 ms 2024-04-02T04:45:05.304 INFO:teuthology.orchestra.run.smithi053.stdout: 5 78378 13.14 MB/sec execute 15 sec latency 33.162 ms 2024-04-02T04:45:05.929 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:05 smithi053 ceph-mon[24889]: pgmap v12: 97 pgs: 97 active+clean; 323 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 30 KiB/s rd, 9.2 MiB/s wr, 891 op/s 2024-04-02T04:45:06.305 INFO:teuthology.orchestra.run.smithi053.stdout: 5 78817 12.83 MB/sec execute 16 sec latency 33.109 ms 2024-04-02T04:45:06.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:05 smithi122 ceph-mon[27145]: pgmap v12: 97 pgs: 97 active+clean; 323 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 30 KiB/s rd, 9.2 MiB/s wr, 891 op/s 2024-04-02T04:45:07.200 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:07 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:07.200 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:07 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:07.200 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:07 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:07.200 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:07 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:07.200 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:07 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:07.200 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:07 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:07.200 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:07 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:07.200 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:07 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:07.200 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:07 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:07.200 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:07 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:07.200 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:07 smithi053 ceph-mon[24889]: Adjusting osd_memory_target on smithi122 to 2704M 2024-04-02T04:45:07.200 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:07 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:07.201 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:07 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:45:07.201 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:07 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:45:07.201 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:07 smithi053 ceph-mon[24889]: Updating smithi053:/etc/ceph/ceph.conf 2024-04-02T04:45:07.201 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:07 smithi053 ceph-mon[24889]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-02T04:45:07.201 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:07 smithi053 ceph-mon[24889]: Updating smithi122:/var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T04:45:07.201 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:07 smithi053 ceph-mon[24889]: Updating smithi053:/var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T04:45:07.306 INFO:teuthology.orchestra.run.smithi053.stdout: 5 79247 12.68 MB/sec execute 17 sec latency 43.287 ms 2024-04-02T04:45:07.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:07 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:07.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:07 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:07.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:07 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:07.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:07 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:07.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:07 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:07.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:07 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:07.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:07 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:07.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:07 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:07.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:07 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:07.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:07 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T04:45:07.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:07 smithi122 ceph-mon[27145]: Adjusting osd_memory_target on smithi122 to 2704M 2024-04-02T04:45:07.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:07 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:07.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:07 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:45:07.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:07 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:45:07.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:07 smithi122 ceph-mon[27145]: Updating smithi053:/etc/ceph/ceph.conf 2024-04-02T04:45:07.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:07 smithi122 ceph-mon[27145]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-02T04:45:07.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:07 smithi122 ceph-mon[27145]: Updating smithi122:/var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T04:45:07.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:07 smithi122 ceph-mon[27145]: Updating smithi053:/var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T04:45:08.208 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:08 smithi053 ceph-mon[24889]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-02T04:45:08.209 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:08 smithi053 ceph-mon[24889]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-04-02T04:45:08.209 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:08 smithi053 ceph-mon[24889]: Updating smithi122:/var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T04:45:08.209 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:08 smithi053 ceph-mon[24889]: pgmap v13: 97 pgs: 97 active+clean; 323 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.8 MiB/s wr, 578 op/s 2024-04-02T04:45:08.209 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:08 smithi053 ceph-mon[24889]: Updating smithi053:/var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T04:45:08.209 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:08 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:08.209 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:08 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:08.209 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:08 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:08.209 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:08 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:08.209 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:08 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:08.209 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:08 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:08.209 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:08 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:08.209 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:08 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:08.209 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:08 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:08.209 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:08 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:08.305 INFO:teuthology.orchestra.run.smithi053.stdout: 5 79752 12.54 MB/sec execute 18 sec latency 46.914 ms 2024-04-02T04:45:08.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:08 smithi122 ceph-mon[27145]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-02T04:45:08.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:08 smithi122 ceph-mon[27145]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-04-02T04:45:08.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:08 smithi122 ceph-mon[27145]: Updating smithi122:/var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T04:45:08.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:08 smithi122 ceph-mon[27145]: pgmap v13: 97 pgs: 97 active+clean; 323 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.8 MiB/s wr, 578 op/s 2024-04-02T04:45:08.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:08 smithi122 ceph-mon[27145]: Updating smithi053:/var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T04:45:08.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:08 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:08.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:08 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:08.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:08 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:08.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:08 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:08.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:08 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:08.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:08 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:08.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:08 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:08.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:08 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:08.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:08 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:08.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:08 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:09.214 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:09 smithi053 ceph-mon[24889]: Reconfiguring alertmanager.smithi053 (dependencies changed)... 2024-04-02T04:45:09.214 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:09 smithi053 ceph-mon[24889]: Reconfiguring daemon alertmanager.smithi053 on smithi053 2024-04-02T04:45:09.306 INFO:teuthology.orchestra.run.smithi053.stdout: 5 80231 12.36 MB/sec execute 19 sec latency 39.553 ms 2024-04-02T04:45:09.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:09 smithi122 ceph-mon[27145]: Reconfiguring alertmanager.smithi053 (dependencies changed)... 2024-04-02T04:45:09.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:09 smithi122 ceph-mon[27145]: Reconfiguring daemon alertmanager.smithi053 on smithi053 2024-04-02T04:45:10.246 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:10 smithi053 ceph-mon[24889]: pgmap v14: 97 pgs: 97 active+clean; 323 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8.9 MiB/s wr, 876 op/s 2024-04-02T04:45:10.305 INFO:teuthology.orchestra.run.smithi053.stdout: 5 80711 12.41 MB/sec execute 20 sec latency 43.420 ms 2024-04-02T04:45:10.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:10 smithi122 ceph-mon[27145]: pgmap v14: 97 pgs: 97 active+clean; 323 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8.9 MiB/s wr, 876 op/s 2024-04-02T04:45:11.305 INFO:teuthology.orchestra.run.smithi053.stdout: 5 81226 12.77 MB/sec execute 21 sec latency 34.494 ms 2024-04-02T04:45:12.287 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:12 smithi053 ceph-mon[24889]: pgmap v15: 97 pgs: 97 active+clean; 323 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 5.9 MiB/s wr, 603 op/s 2024-04-02T04:45:12.305 INFO:teuthology.orchestra.run.smithi053.stdout: 5 81757 13.17 MB/sec execute 22 sec latency 30.650 ms 2024-04-02T04:45:12.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:12 smithi122 ceph-mon[27145]: pgmap v15: 97 pgs: 97 active+clean; 323 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 5.9 MiB/s wr, 603 op/s 2024-04-02T04:45:13.305 INFO:teuthology.orchestra.run.smithi053.stdout: 5 82125 12.92 MB/sec execute 23 sec latency 41.103 ms 2024-04-02T04:45:14.305 INFO:teuthology.orchestra.run.smithi053.stdout: 5 82492 12.59 MB/sec execute 24 sec latency 27.390 ms 2024-04-02T04:45:14.306 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:14 smithi053 ceph-mon[24889]: pgmap v16: 97 pgs: 97 active+clean; 323 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 5.9 MiB/s wr, 603 op/s 2024-04-02T04:45:14.446 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:14 smithi122 ceph-mon[27145]: pgmap v16: 97 pgs: 97 active+clean; 323 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 5.9 MiB/s wr, 603 op/s 2024-04-02T04:45:15.305 INFO:teuthology.orchestra.run.smithi053.stdout: 5 82995 12.53 MB/sec execute 25 sec latency 43.655 ms 2024-04-02T04:45:16.305 INFO:teuthology.orchestra.run.smithi053.stdout: 5 83642 12.60 MB/sec execute 26 sec latency 39.872 ms 2024-04-02T04:45:16.317 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:16 smithi053 ceph-mon[24889]: pgmap v17: 97 pgs: 97 active+clean; 332 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.1 MiB/s wr, 883 op/s 2024-04-02T04:45:16.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:16 smithi122 ceph-mon[27145]: pgmap v17: 97 pgs: 97 active+clean; 332 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.1 MiB/s wr, 883 op/s 2024-04-02T04:45:17.305 INFO:teuthology.orchestra.run.smithi053.stdout: 5 84214 12.69 MB/sec execute 27 sec latency 89.613 ms 2024-04-02T04:45:17.322 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:17 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:45:17.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:17 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:45:18.305 INFO:teuthology.orchestra.run.smithi053.stdout: 5 84953 13.34 MB/sec execute 28 sec latency 29.538 ms 2024-04-02T04:45:18.327 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:18 smithi053 ceph-mon[24889]: pgmap v18: 97 pgs: 97 active+clean; 332 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.4 MiB/s wr, 577 op/s 2024-04-02T04:45:18.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:18 smithi122 ceph-mon[27145]: pgmap v18: 97 pgs: 97 active+clean; 332 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.4 MiB/s wr, 577 op/s 2024-04-02T04:45:19.306 INFO:teuthology.orchestra.run.smithi053.stdout: 5 85461 13.39 MB/sec execute 29 sec latency 25.709 ms 2024-04-02T04:45:20.305 INFO:teuthology.orchestra.run.smithi053.stdout: 5 85929 13.21 MB/sec execute 30 sec latency 35.116 ms 2024-04-02T04:45:20.352 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:20 smithi053 ceph-mon[24889]: pgmap v19: 97 pgs: 97 active+clean; 330 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 9.5 MiB/s wr, 904 op/s 2024-04-02T04:45:20.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:20 smithi122 ceph-mon[27145]: pgmap v19: 97 pgs: 97 active+clean; 330 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 9.5 MiB/s wr, 904 op/s 2024-04-02T04:45:21.305 INFO:teuthology.orchestra.run.smithi053.stdout: 5 86414 13.17 MB/sec execute 31 sec latency 45.100 ms 2024-04-02T04:45:22.306 INFO:teuthology.orchestra.run.smithi053.stdout: 5 86951 13.08 MB/sec execute 32 sec latency 50.129 ms 2024-04-02T04:45:22.363 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:22 smithi053 ceph-mon[24889]: pgmap v20: 97 pgs: 97 active+clean; 330 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.3 MiB/s wr, 605 op/s 2024-04-02T04:45:22.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:22 smithi122 ceph-mon[27145]: pgmap v20: 97 pgs: 97 active+clean; 330 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.3 MiB/s wr, 605 op/s 2024-04-02T04:45:23.306 INFO:teuthology.orchestra.run.smithi053.stdout: 5 87532 13.09 MB/sec execute 33 sec latency 31.002 ms 2024-04-02T04:45:24.306 INFO:teuthology.orchestra.run.smithi053.stdout: 5 88151 13.41 MB/sec execute 34 sec latency 38.252 ms 2024-04-02T04:45:24.402 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:24 smithi053 ceph-mon[24889]: pgmap v21: 97 pgs: 97 active+clean; 330 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.3 MiB/s wr, 605 op/s 2024-04-02T04:45:24.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:24 smithi122 ceph-mon[27145]: pgmap v21: 97 pgs: 97 active+clean; 330 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.3 MiB/s wr, 605 op/s 2024-04-02T04:45:25.306 INFO:teuthology.orchestra.run.smithi053.stdout: 5 88632 13.60 MB/sec execute 35 sec latency 31.774 ms 2024-04-02T04:45:25.915 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:25 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:25.916 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:25 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:25.916 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:25 smithi053 ceph-mon[24889]: Reconfiguring grafana.smithi053 (dependencies changed)... 2024-04-02T04:45:25.916 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:25 smithi053 ceph-mon[24889]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-02T04:45:25.916 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:25 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:25.916 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:25 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:25.916 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:25 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-02T04:45:25.916 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:25 smithi053 ceph-mon[24889]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-02T04:45:25.916 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:25 smithi053 ceph-mon[24889]: Reconfiguring daemon grafana.smithi053 on smithi053 2024-04-02T04:45:25.916 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:25 smithi053 ceph-mon[24889]: pgmap v22: 97 pgs: 97 active+clean; 320 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.3 MiB/s wr, 926 op/s 2024-04-02T04:45:26.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:25 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:26.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:25 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:26.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:25 smithi122 ceph-mon[27145]: Reconfiguring grafana.smithi053 (dependencies changed)... 2024-04-02T04:45:26.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:25 smithi122 ceph-mon[27145]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-02T04:45:26.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:25 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:26.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:25 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:26.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:25 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-02T04:45:26.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:25 smithi122 ceph-mon[27145]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-02T04:45:26.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:25 smithi122 ceph-mon[27145]: Reconfiguring daemon grafana.smithi053 on smithi053 2024-04-02T04:45:26.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:25 smithi122 ceph-mon[27145]: pgmap v22: 97 pgs: 97 active+clean; 320 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.3 MiB/s wr, 926 op/s 2024-04-02T04:45:26.307 INFO:teuthology.orchestra.run.smithi053.stdout: 5 89000 13.44 MB/sec execute 36 sec latency 39.887 ms 2024-04-02T04:45:27.307 INFO:teuthology.orchestra.run.smithi053.stdout: 5 89369 13.24 MB/sec execute 37 sec latency 43.816 ms 2024-04-02T04:45:28.002 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:27 smithi053 ceph-mon[24889]: pgmap v23: 97 pgs: 97 active+clean; 320 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 6.1 MiB/s wr, 647 op/s 2024-04-02T04:45:28.307 INFO:teuthology.orchestra.run.smithi053.stdout: 5 89701 13.10 MB/sec execute 38 sec latency 32.211 ms 2024-04-02T04:45:28.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:27 smithi122 ceph-mon[27145]: pgmap v23: 97 pgs: 97 active+clean; 320 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 6.1 MiB/s wr, 647 op/s 2024-04-02T04:45:28.841 INFO:teuthology.orchestra.run.smithi053.stdout:true 2024-04-02T04:45:29.307 INFO:teuthology.orchestra.run.smithi053.stdout: 5 90177 13.09 MB/sec execute 39 sec latency 33.321 ms 2024-04-02T04:45:30.035 INFO:teuthology.orchestra.run.smithi053.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-02T04:45:30.035 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager.smithi053 smithi053 *:9093,9094 starting - - - - 2024-04-02T04:45:30.036 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi053 smithi053 running (9m) 28s ago 9m 6409k - 17.2.0 e1d6a67b021e a5c436bbd36d 2024-04-02T04:45:30.036 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi122 smithi122 running (8m) 29s ago 8m 6405k - 17.2.0 e1d6a67b021e b7074474cd46 2024-04-02T04:45:30.036 INFO:teuthology.orchestra.run.smithi053.stdout:grafana.smithi053 smithi053 *:3000 running (7m) 28s ago 8m 53.9M - 8.3.5 dad864ee21e9 8f4a8d871d62 2024-04-02T04:45:30.036 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi053.nfrfot smithi053 running (3m) 28s ago 3m 35.8M - 17.2.0 e1d6a67b021e c94e9e643514 2024-04-02T04:45:30.036 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi122.mwoyoh smithi122 running (3m) 29s ago 3m 14.1M - 17.2.0 e1d6a67b021e 73376d9ee4b8 2024-04-02T04:45:30.036 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi053.legdmv smithi053 *:9283 running (10m) 28s ago 10m 328M - 17.2.0 e1d6a67b021e 5616241666e4 2024-04-02T04:45:30.036 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi122.cpwlyk smithi122 *:8443,9283 running (53s) 29s ago 8m 489M - 19.0.0-1979-g6cec793d 78f830190839 3a534988cf3b 2024-04-02T04:45:30.036 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi053 smithi053 running (10m) 28s ago 10m 51.9M 2048M 17.2.0 e1d6a67b021e 665e4412f6da 2024-04-02T04:45:30.036 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi122 smithi122 running (8m) 29s ago 8m 40.9M 2048M 17.2.0 e1d6a67b021e a2df07e0883d 2024-04-02T04:45:30.036 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.0.0.smithi053.mhkdcb smithi053 *:2049 running (2m) 28s ago 2m 250M - 4.0 e1d6a67b021e 07e5da7bd2c5 2024-04-02T04:45:30.036 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.1.0.smithi122.mauvsb smithi122 *:2049 running (2m) 29s ago 2m 73.4M - 4.0 e1d6a67b021e 4f866685005e 2024-04-02T04:45:30.036 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi053 smithi053 *:9100 running (8m) 28s ago 8m 17.4M - 1.3.1 1dbe0e931976 87185e572b31 2024-04-02T04:45:30.036 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi122 smithi122 *:9100 running (8m) 29s ago 8m 19.7M - 1.3.1 1dbe0e931976 d576cf080067 2024-04-02T04:45:30.036 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 smithi053 running (7m) 28s ago 7m 299M 1937M 17.2.0 e1d6a67b021e 906cb1fd8042 2024-04-02T04:45:30.037 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 smithi053 running (7m) 28s ago 7m 238M 1937M 17.2.0 e1d6a67b021e ac1ad6266e63 2024-04-02T04:45:30.037 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 smithi053 running (6m) 28s ago 6m 208M 1937M 17.2.0 e1d6a67b021e 86964c8a94ff 2024-04-02T04:45:30.037 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 smithi053 running (6m) 28s ago 6m 252M 1937M 17.2.0 e1d6a67b021e 7e8b9e98ae2b 2024-04-02T04:45:30.037 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 smithi122 running (5m) 29s ago 5m 259M 2704M 17.2.0 e1d6a67b021e 46076afcd7bc 2024-04-02T04:45:30.037 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 smithi122 running (5m) 29s ago 5m 290M 2704M 17.2.0 e1d6a67b021e e687a2f6da4d 2024-04-02T04:45:30.037 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 smithi122 running (5m) 29s ago 5m 261M 2704M 17.2.0 e1d6a67b021e 9e371df9a839 2024-04-02T04:45:30.037 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 smithi122 running (4m) 29s ago 4m 353M 2704M 17.2.0 e1d6a67b021e 9b3cc398254d 2024-04-02T04:45:30.037 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus.smithi053 smithi053 *:9095 running (7m) 28s ago 7m 64.3M - 2.33.4 514e6a882f6e 36d493ce400e 2024-04-02T04:45:30.275 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:30 smithi053 ceph-mon[24889]: from='client.24511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:45:30.275 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:30 smithi053 ceph-mon[24889]: pgmap v24: 97 pgs: 97 active+clean; 330 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.3 MiB/s wr, 911 op/s 2024-04-02T04:45:30.275 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:30 smithi053 ceph-mon[24889]: from='client.14716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:45:30.307 INFO:teuthology.orchestra.run.smithi053.stdout: 5 90581 12.86 MB/sec execute 40 sec latency 56.937 ms 2024-04-02T04:45:30.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:30 smithi122 ceph-mon[27145]: from='client.24511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:45:30.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:30 smithi122 ceph-mon[27145]: pgmap v24: 97 pgs: 97 active+clean; 330 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.3 MiB/s wr, 911 op/s 2024-04-02T04:45:30.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:30 smithi122 ceph-mon[27145]: from='client.14716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:45:30.765 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:45:30.765 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": { 2024-04-02T04:45:30.765 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:45:30.765 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:45:30.766 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": { 2024-04-02T04:45:30.766 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-02T04:45:30.766 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 1 2024-04-02T04:45:30.766 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:45:30.766 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": { 2024-04-02T04:45:30.766 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-02T04:45:30.766 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:45:30.766 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": { 2024-04-02T04:45:30.766 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:45:30.766 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:45:30.766 INFO:teuthology.orchestra.run.smithi053.stdout: "overall": { 2024-04-02T04:45:30.766 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-04-02T04:45:30.766 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 1 2024-04-02T04:45:30.766 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-02T04:45:30.766 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:45:31.280 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:31 smithi053 ceph-mon[24889]: from='client.14720 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:45:31.280 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:31 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/1263369299' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:45:31.309 INFO:teuthology.orchestra.run.smithi053.stdout: 5 90998 12.83 MB/sec execute 41 sec latency 48.352 ms 2024-04-02T04:45:31.352 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:45:31.353 INFO:teuthology.orchestra.run.smithi053.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae", 2024-04-02T04:45:31.353 INFO:teuthology.orchestra.run.smithi053.stdout: "in_progress": true, 2024-04-02T04:45:31.353 INFO:teuthology.orchestra.run.smithi053.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-02T04:45:31.353 INFO:teuthology.orchestra.run.smithi053.stdout: "services_complete": [], 2024-04-02T04:45:31.353 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": "1/23 daemons upgraded", 2024-04-02T04:45:31.353 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "", 2024-04-02T04:45:31.353 INFO:teuthology.orchestra.run.smithi053.stdout: "is_paused": false 2024-04-02T04:45:31.353 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:45:31.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:31 smithi122 ceph-mon[27145]: from='client.14720 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:45:31.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:31 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/1263369299' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:45:32.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:32 smithi053 ceph-mon[24889]: pgmap v25: 97 pgs: 97 active+clean; 330 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.1 MiB/s wr, 584 op/s 2024-04-02T04:45:32.043 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:32 smithi053 ceph-mon[24889]: from='client.24519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:45:32.044 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:32 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:45:32.045 INFO:teuthology.orchestra.run.smithi053.stdout:HEALTH_OK 2024-04-02T04:45:32.309 INFO:teuthology.orchestra.run.smithi053.stdout: 5 91563 12.97 MB/sec execute 42 sec latency 32.665 ms 2024-04-02T04:45:32.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:32 smithi122 ceph-mon[27145]: pgmap v25: 97 pgs: 97 active+clean; 330 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.1 MiB/s wr, 584 op/s 2024-04-02T04:45:32.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:32 smithi122 ceph-mon[27145]: from='client.24519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:45:32.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:32 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:45:33.056 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:33 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/817123568' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:45:33.310 INFO:teuthology.orchestra.run.smithi053.stdout: 5 92048 13.08 MB/sec execute 43 sec latency 26.053 ms 2024-04-02T04:45:33.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:33 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/817123568' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:45:34.063 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:34 smithi053 ceph-mon[24889]: pgmap v26: 97 pgs: 97 active+clean; 330 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.1 MiB/s wr, 584 op/s 2024-04-02T04:45:34.309 INFO:teuthology.orchestra.run.smithi053.stdout: 5 92493 13.14 MB/sec execute 44 sec latency 31.637 ms 2024-04-02T04:45:34.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:34 smithi122 ceph-mon[27145]: pgmap v26: 97 pgs: 97 active+clean; 330 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.1 MiB/s wr, 584 op/s 2024-04-02T04:45:35.309 INFO:teuthology.orchestra.run.smithi053.stdout: 5 92951 13.04 MB/sec execute 45 sec latency 29.495 ms 2024-04-02T04:45:36.086 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:36 smithi053 ceph-mon[24889]: pgmap v27: 97 pgs: 97 active+clean; 327 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.9 MiB/s wr, 856 op/s 2024-04-02T04:45:36.309 INFO:teuthology.orchestra.run.smithi053.stdout: 5 93341 12.95 MB/sec execute 46 sec latency 44.882 ms 2024-04-02T04:45:36.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:36 smithi122 ceph-mon[27145]: pgmap v27: 97 pgs: 97 active+clean; 327 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.9 MiB/s wr, 856 op/s 2024-04-02T04:45:37.309 INFO:teuthology.orchestra.run.smithi053.stdout: 5 93819 12.91 MB/sec execute 47 sec latency 64.022 ms 2024-04-02T04:45:38.128 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:38 smithi053 ceph-mon[24889]: pgmap v28: 97 pgs: 97 active+clean; 327 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.9 MiB/s wr, 535 op/s 2024-04-02T04:45:38.309 INFO:teuthology.orchestra.run.smithi053.stdout: 5 94274 12.76 MB/sec execute 48 sec latency 52.409 ms 2024-04-02T04:45:38.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:38 smithi122 ceph-mon[27145]: pgmap v28: 97 pgs: 97 active+clean; 327 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.9 MiB/s wr, 535 op/s 2024-04-02T04:45:39.309 INFO:teuthology.orchestra.run.smithi053.stdout: 5 94663 12.75 MB/sec execute 49 sec latency 29.493 ms 2024-04-02T04:45:40.150 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:40 smithi053 ceph-mon[24889]: pgmap v29: 97 pgs: 97 active+clean; 325 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.4 MiB/s wr, 806 op/s 2024-04-02T04:45:40.309 INFO:teuthology.orchestra.run.smithi053.stdout: 5 95118 12.81 MB/sec execute 50 sec latency 46.562 ms 2024-04-02T04:45:40.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:40 smithi122 ceph-mon[27145]: pgmap v29: 97 pgs: 97 active+clean; 325 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.4 MiB/s wr, 806 op/s 2024-04-02T04:45:41.309 INFO:teuthology.orchestra.run.smithi053.stdout: 5 95528 12.88 MB/sec execute 51 sec latency 45.284 ms 2024-04-02T04:45:42.202 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:42 smithi053 ceph-mon[24889]: pgmap v30: 97 pgs: 97 active+clean; 325 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.2 MiB/s wr, 542 op/s 2024-04-02T04:45:42.309 INFO:teuthology.orchestra.run.smithi053.stdout: 5 95908 12.93 MB/sec execute 52 sec latency 34.523 ms 2024-04-02T04:45:42.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:42 smithi122 ceph-mon[27145]: pgmap v30: 97 pgs: 97 active+clean; 325 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.2 MiB/s wr, 542 op/s 2024-04-02T04:45:43.310 INFO:teuthology.orchestra.run.smithi053.stdout: 5 96251 12.81 MB/sec execute 53 sec latency 40.191 ms 2024-04-02T04:45:44.267 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:44 smithi053 ceph-mon[24889]: pgmap v31: 97 pgs: 97 active+clean; 325 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.2 MiB/s wr, 542 op/s 2024-04-02T04:45:44.309 INFO:teuthology.orchestra.run.smithi053.stdout: 5 96609 12.66 MB/sec execute 54 sec latency 53.845 ms 2024-04-02T04:45:44.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:44 smithi122 ceph-mon[27145]: pgmap v31: 97 pgs: 97 active+clean; 325 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.2 MiB/s wr, 542 op/s 2024-04-02T04:45:45.309 INFO:teuthology.orchestra.run.smithi053.stdout: 5 97052 12.63 MB/sec execute 55 sec latency 34.576 ms 2024-04-02T04:45:46.298 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:46 smithi053 ceph-mon[24889]: pgmap v32: 97 pgs: 97 active+clean; 330 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.1 MiB/s wr, 777 op/s 2024-04-02T04:45:46.309 INFO:teuthology.orchestra.run.smithi053.stdout: 5 97603 12.58 MB/sec execute 56 sec latency 46.155 ms 2024-04-02T04:45:46.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:46 smithi122 ceph-mon[27145]: pgmap v32: 97 pgs: 97 active+clean; 330 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.1 MiB/s wr, 777 op/s 2024-04-02T04:45:47.310 INFO:teuthology.orchestra.run.smithi053.stdout: 5 98056 12.54 MB/sec execute 57 sec latency 43.742 ms 2024-04-02T04:45:47.822 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:47 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:47.823 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:47 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:47.823 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:47 smithi053 ceph-mon[24889]: Reconfiguring node-exporter.smithi053 (dependencies changed)... 2024-04-02T04:45:47.823 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:47 smithi053 ceph-mon[24889]: Reconfiguring daemon node-exporter.smithi053 on smithi053 2024-04-02T04:45:47.823 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:47 smithi053 ceph-mon[24889]: pgmap v33: 97 pgs: 97 active+clean; 330 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.4 MiB/s wr, 505 op/s 2024-04-02T04:45:47.823 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:47 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:45:48.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:47 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:48.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:47 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:48.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:47 smithi122 ceph-mon[27145]: Reconfiguring node-exporter.smithi053 (dependencies changed)... 2024-04-02T04:45:48.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:47 smithi122 ceph-mon[27145]: Reconfiguring daemon node-exporter.smithi053 on smithi053 2024-04-02T04:45:48.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:47 smithi122 ceph-mon[27145]: pgmap v33: 97 pgs: 97 active+clean; 330 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.4 MiB/s wr, 505 op/s 2024-04-02T04:45:48.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:47 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:45:48.310 INFO:teuthology.orchestra.run.smithi053.stdout: 5 98439 12.54 MB/sec execute 58 sec latency 61.112 ms 2024-04-02T04:45:49.310 INFO:teuthology.orchestra.run.smithi053.stdout: 5 98879 12.61 MB/sec execute 59 sec latency 48.658 ms 2024-04-02T04:45:50.117 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:50 smithi053 ceph-mon[24889]: pgmap v34: 97 pgs: 97 active+clean; 321 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.7 MiB/s wr, 781 op/s 2024-04-02T04:45:50.310 INFO:teuthology.orchestra.run.smithi053.stdout: 5 99302 12.68 MB/sec execute 60 sec latency 32.386 ms 2024-04-02T04:45:50.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:50 smithi122 ceph-mon[27145]: pgmap v34: 97 pgs: 97 active+clean; 321 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.7 MiB/s wr, 781 op/s 2024-04-02T04:45:51.310 INFO:teuthology.orchestra.run.smithi053.stdout: 5 99633 12.62 MB/sec execute 61 sec latency 43.612 ms 2024-04-02T04:45:52.152 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:52 smithi053 ceph-mon[24889]: pgmap v35: 97 pgs: 97 active+clean; 321 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.2 MiB/s wr, 510 op/s 2024-04-02T04:45:52.152 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:52 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:52.152 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:52 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:52.310 INFO:teuthology.orchestra.run.smithi053.stdout: 5 99921 12.52 MB/sec execute 62 sec latency 38.592 ms 2024-04-02T04:45:52.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:52 smithi122 ceph-mon[27145]: pgmap v35: 97 pgs: 97 active+clean; 321 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.2 MiB/s wr, 510 op/s 2024-04-02T04:45:52.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:52 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:52.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:52 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:45:53.176 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:53 smithi053 ceph-mon[24889]: Reconfiguring prometheus.smithi053 (dependencies changed)... 2024-04-02T04:45:53.176 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:53 smithi053 ceph-mon[24889]: Reconfiguring daemon prometheus.smithi053 on smithi053 2024-04-02T04:45:53.310 INFO:teuthology.orchestra.run.smithi053.stdout: 5 100206 12.38 MB/sec execute 63 sec latency 52.979 ms 2024-04-02T04:45:53.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:53 smithi122 ceph-mon[27145]: Reconfiguring prometheus.smithi053 (dependencies changed)... 2024-04-02T04:45:53.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:53 smithi122 ceph-mon[27145]: Reconfiguring daemon prometheus.smithi053 on smithi053 2024-04-02T04:45:54.028 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:54 smithi053 ceph-mon[24889]: pgmap v36: 97 pgs: 97 active+clean; 321 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.2 MiB/s wr, 510 op/s 2024-04-02T04:45:54.310 INFO:teuthology.orchestra.run.smithi053.stdout: 5 100506 12.30 MB/sec execute 64 sec latency 42.853 ms 2024-04-02T04:45:54.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:54 smithi122 ceph-mon[27145]: pgmap v36: 97 pgs: 97 active+clean; 321 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.2 MiB/s wr, 510 op/s 2024-04-02T04:45:55.310 INFO:teuthology.orchestra.run.smithi053.stdout: 5 100896 12.28 MB/sec execute 65 sec latency 56.633 ms 2024-04-02T04:45:56.071 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:56 smithi053 ceph-mon[24889]: pgmap v37: 97 pgs: 97 active+clean; 332 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.8 MiB/s wr, 724 op/s 2024-04-02T04:45:56.310 INFO:teuthology.orchestra.run.smithi053.stdout: 5 101309 12.16 MB/sec execute 66 sec latency 41.629 ms 2024-04-02T04:45:56.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:56 smithi122 ceph-mon[27145]: pgmap v37: 97 pgs: 97 active+clean; 332 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.8 MiB/s wr, 724 op/s 2024-04-02T04:45:57.310 INFO:teuthology.orchestra.run.smithi053.stdout: 5 101687 12.17 MB/sec execute 67 sec latency 47.762 ms 2024-04-02T04:45:58.089 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:45:58 smithi053 ceph-mon[24889]: pgmap v38: 97 pgs: 97 active+clean; 332 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.9 MiB/s wr, 489 op/s 2024-04-02T04:45:58.310 INFO:teuthology.orchestra.run.smithi053.stdout: 5 102022 12.11 MB/sec execute 68 sec latency 79.794 ms 2024-04-02T04:45:58.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:45:58 smithi122 ceph-mon[27145]: pgmap v38: 97 pgs: 97 active+clean; 332 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.9 MiB/s wr, 489 op/s 2024-04-02T04:45:59.310 INFO:teuthology.orchestra.run.smithi053.stdout: 5 102417 12.17 MB/sec execute 69 sec latency 34.640 ms 2024-04-02T04:46:00.137 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:00 smithi053 ceph-mon[24889]: pgmap v39: 97 pgs: 97 active+clean; 323 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 6.8 MiB/s wr, 715 op/s 2024-04-02T04:46:00.310 INFO:teuthology.orchestra.run.smithi053.stdout: 5 102737 12.18 MB/sec execute 70 sec latency 58.563 ms 2024-04-02T04:46:00.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:00 smithi122 ceph-mon[27145]: pgmap v39: 97 pgs: 97 active+clean; 323 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 6.8 MiB/s wr, 715 op/s 2024-04-02T04:46:01.310 INFO:teuthology.orchestra.run.smithi053.stdout: 5 103076 12.17 MB/sec execute 71 sec latency 71.400 ms 2024-04-02T04:46:02.154 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:02 smithi053 ceph-mon[24889]: pgmap v40: 97 pgs: 97 active+clean; 323 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.5 MiB/s wr, 440 op/s 2024-04-02T04:46:02.155 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:02 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:46:02.310 INFO:teuthology.orchestra.run.smithi053.stdout: 5 103422 12.10 MB/sec execute 72 sec latency 41.205 ms 2024-04-02T04:46:02.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:02 smithi122 ceph-mon[27145]: pgmap v40: 97 pgs: 97 active+clean; 323 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.5 MiB/s wr, 440 op/s 2024-04-02T04:46:02.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:02 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:46:02.618 INFO:teuthology.orchestra.run.smithi053.stdout:true 2024-04-02T04:46:03.310 INFO:teuthology.orchestra.run.smithi053.stdout: 5 103810 12.03 MB/sec execute 73 sec latency 45.391 ms 2024-04-02T04:46:03.724 INFO:teuthology.orchestra.run.smithi053.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-02T04:46:03.725 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager.smithi053 smithi053 *:9093,9094 starting - - - - 2024-04-02T04:46:03.725 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi053 smithi053 running (9m) 62s ago 9m 6409k - 17.2.0 e1d6a67b021e a5c436bbd36d 2024-04-02T04:46:03.725 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi122 smithi122 running (9m) 62s ago 8m 6405k - 17.2.0 e1d6a67b021e b7074474cd46 2024-04-02T04:46:03.725 INFO:teuthology.orchestra.run.smithi053.stdout:grafana.smithi053 smithi053 *:3000 starting - - - - 2024-04-02T04:46:03.725 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi053.nfrfot smithi053 running (3m) 62s ago 3m 35.8M - 17.2.0 e1d6a67b021e c94e9e643514 2024-04-02T04:46:03.725 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi122.mwoyoh smithi122 running (3m) 62s ago 3m 14.1M - 17.2.0 e1d6a67b021e 73376d9ee4b8 2024-04-02T04:46:03.725 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi053.legdmv smithi053 *:9283 running (10m) 62s ago 10m 328M - 17.2.0 e1d6a67b021e 5616241666e4 2024-04-02T04:46:03.725 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi122.cpwlyk smithi122 *:8443,9283 running (87s) 62s ago 8m 489M - 19.0.0-1979-g6cec793d 78f830190839 3a534988cf3b 2024-04-02T04:46:03.725 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi053 smithi053 running (11m) 62s ago 11m 51.9M 2048M 17.2.0 e1d6a67b021e 665e4412f6da 2024-04-02T04:46:03.725 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi122 smithi122 running (8m) 62s ago 8m 40.9M 2048M 17.2.0 e1d6a67b021e a2df07e0883d 2024-04-02T04:46:03.726 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.0.0.smithi053.mhkdcb smithi053 *:2049 running (3m) 62s ago 3m 250M - 4.0 e1d6a67b021e 07e5da7bd2c5 2024-04-02T04:46:03.726 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.1.0.smithi122.mauvsb smithi122 *:2049 running (3m) 62s ago 3m 73.4M - 4.0 e1d6a67b021e 4f866685005e 2024-04-02T04:46:03.726 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi053 smithi053 *:9100 starting - - - - 2024-04-02T04:46:03.726 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi122 smithi122 *:9100 running (8m) 62s ago 8m 19.7M - 1.3.1 1dbe0e931976 d576cf080067 2024-04-02T04:46:03.726 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 smithi053 running (8m) 62s ago 8m 299M 1937M 17.2.0 e1d6a67b021e 906cb1fd8042 2024-04-02T04:46:03.726 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 smithi053 running (7m) 62s ago 7m 238M 1937M 17.2.0 e1d6a67b021e ac1ad6266e63 2024-04-02T04:46:03.726 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 smithi053 running (7m) 62s ago 7m 208M 1937M 17.2.0 e1d6a67b021e 86964c8a94ff 2024-04-02T04:46:03.726 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 smithi053 running (6m) 62s ago 6m 252M 1937M 17.2.0 e1d6a67b021e 7e8b9e98ae2b 2024-04-02T04:46:03.726 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 smithi122 running (6m) 62s ago 6m 259M 2704M 17.2.0 e1d6a67b021e 46076afcd7bc 2024-04-02T04:46:03.726 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 smithi122 running (6m) 62s ago 6m 290M 2704M 17.2.0 e1d6a67b021e e687a2f6da4d 2024-04-02T04:46:03.726 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 smithi122 running (5m) 62s ago 5m 261M 2704M 17.2.0 e1d6a67b021e 9e371df9a839 2024-04-02T04:46:03.726 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 smithi122 running (5m) 62s ago 5m 353M 2704M 17.2.0 e1d6a67b021e 9b3cc398254d 2024-04-02T04:46:03.726 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus.smithi053 smithi053 *:9095 running (8m) 62s ago 8m 64.3M - 2.33.4 514e6a882f6e 36d493ce400e 2024-04-02T04:46:04.169 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:04 smithi053 ceph-mon[24889]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:46:04.170 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:04 smithi053 ceph-mon[24889]: pgmap v41: 97 pgs: 97 active+clean; 323 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.5 MiB/s wr, 440 op/s 2024-04-02T04:46:04.170 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:04 smithi053 ceph-mon[24889]: from='client.14740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:46:04.310 INFO:teuthology.orchestra.run.smithi053.stdout: 5 104237 11.98 MB/sec execute 74 sec latency 30.850 ms 2024-04-02T04:46:04.380 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:46:04.380 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": { 2024-04-02T04:46:04.380 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:46:04.380 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:46:04.381 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": { 2024-04-02T04:46:04.381 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-02T04:46:04.381 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 1 2024-04-02T04:46:04.381 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:46:04.381 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": { 2024-04-02T04:46:04.381 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-02T04:46:04.381 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:46:04.382 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": { 2024-04-02T04:46:04.382 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:46:04.382 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:46:04.382 INFO:teuthology.orchestra.run.smithi053.stdout: "overall": { 2024-04-02T04:46:04.382 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-04-02T04:46:04.382 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 1 2024-04-02T04:46:04.382 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-02T04:46:04.383 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:46:04.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:04 smithi122 ceph-mon[27145]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:46:04.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:04 smithi122 ceph-mon[27145]: pgmap v41: 97 pgs: 97 active+clean; 323 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.5 MiB/s wr, 440 op/s 2024-04-02T04:46:04.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:04 smithi122 ceph-mon[27145]: from='client.14740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:46:04.884 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:46:04.884 INFO:teuthology.orchestra.run.smithi053.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae", 2024-04-02T04:46:04.884 INFO:teuthology.orchestra.run.smithi053.stdout: "in_progress": true, 2024-04-02T04:46:04.885 INFO:teuthology.orchestra.run.smithi053.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-02T04:46:04.885 INFO:teuthology.orchestra.run.smithi053.stdout: "services_complete": [], 2024-04-02T04:46:04.885 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": "1/23 daemons upgraded", 2024-04-02T04:46:04.885 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "", 2024-04-02T04:46:04.885 INFO:teuthology.orchestra.run.smithi053.stdout: "is_paused": false 2024-04-02T04:46:04.885 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:46:05.199 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:05 smithi053 ceph-mon[24889]: from='client.14744 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:46:05.200 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:05 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/2451124861' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:46:05.310 INFO:teuthology.orchestra.run.smithi053.stdout: 5 104624 11.94 MB/sec execute 75 sec latency 146.507 ms 2024-04-02T04:46:05.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:05 smithi122 ceph-mon[27145]: from='client.14744 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:46:05.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:05 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/2451124861' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:46:05.520 INFO:teuthology.orchestra.run.smithi053.stdout:HEALTH_OK 2024-04-02T04:46:06.204 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:06 smithi053 ceph-mon[24889]: from='client.24539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:46:06.204 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:06 smithi053 ceph-mon[24889]: pgmap v42: 97 pgs: 97 active+clean; 332 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.4 KiB/s rd, 7.0 MiB/s wr, 645 op/s 2024-04-02T04:46:06.204 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:06 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/3778213148' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:46:06.310 INFO:teuthology.orchestra.run.smithi053.stdout: 5 105161 11.93 MB/sec execute 76 sec latency 31.803 ms 2024-04-02T04:46:06.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:06 smithi122 ceph-mon[27145]: from='client.24539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:46:06.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:06 smithi122 ceph-mon[27145]: pgmap v42: 97 pgs: 97 active+clean; 332 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.4 KiB/s rd, 7.0 MiB/s wr, 645 op/s 2024-04-02T04:46:06.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:06 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/3778213148' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:46:07.313 INFO:teuthology.orchestra.run.smithi053.stdout: 5 105579 11.94 MB/sec execute 77 sec latency 34.150 ms 2024-04-02T04:46:08.251 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:08 smithi053 ceph-mon[24889]: pgmap v43: 97 pgs: 97 active+clean; 332 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.4 MiB/s wr, 431 op/s 2024-04-02T04:46:08.313 INFO:teuthology.orchestra.run.smithi053.stdout: 5 106096 12.03 MB/sec execute 78 sec latency 25.187 ms 2024-04-02T04:46:08.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:08 smithi122 ceph-mon[27145]: pgmap v43: 97 pgs: 97 active+clean; 332 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.4 MiB/s wr, 431 op/s 2024-04-02T04:46:09.313 INFO:teuthology.orchestra.run.smithi053.stdout: 5 106484 12.10 MB/sec execute 79 sec latency 40.685 ms 2024-04-02T04:46:10.051 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:10 smithi053 ceph-mon[24889]: pgmap v44: 97 pgs: 97 active+clean; 329 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 6.9 MiB/s wr, 699 op/s 2024-04-02T04:46:10.313 INFO:teuthology.orchestra.run.smithi053.stdout: 5 106744 12.02 MB/sec execute 80 sec latency 73.910 ms 2024-04-02T04:46:10.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:10 smithi122 ceph-mon[27145]: pgmap v44: 97 pgs: 97 active+clean; 329 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 6.9 MiB/s wr, 699 op/s 2024-04-02T04:46:11.313 INFO:teuthology.orchestra.run.smithi053.stdout: 5 107268 12.02 MB/sec execute 81 sec latency 72.651 ms 2024-04-02T04:46:12.061 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:12 smithi053 ceph-mon[24889]: pgmap v45: 97 pgs: 97 active+clean; 329 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.0 MiB/s wr, 473 op/s 2024-04-02T04:46:12.313 INFO:teuthology.orchestra.run.smithi053.stdout: 5 107979 12.07 MB/sec execute 82 sec latency 51.970 ms 2024-04-02T04:46:12.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:12 smithi122 ceph-mon[27145]: pgmap v45: 97 pgs: 97 active+clean; 329 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.0 MiB/s wr, 473 op/s 2024-04-02T04:46:13.313 INFO:teuthology.orchestra.run.smithi053.stdout: 5 108774 12.12 MB/sec execute 83 sec latency 30.367 ms 2024-04-02T04:46:14.091 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:14 smithi053 ceph-mon[24889]: pgmap v46: 97 pgs: 97 active+clean; 329 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.0 MiB/s wr, 473 op/s 2024-04-02T04:46:14.313 INFO:teuthology.orchestra.run.smithi053.stdout: 5 109332 12.21 MB/sec execute 84 sec latency 39.073 ms 2024-04-02T04:46:14.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:14 smithi122 ceph-mon[27145]: pgmap v46: 97 pgs: 97 active+clean; 329 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.0 MiB/s wr, 473 op/s 2024-04-02T04:46:15.313 INFO:teuthology.orchestra.run.smithi053.stdout: 5 109876 12.28 MB/sec execute 85 sec latency 26.744 ms 2024-04-02T04:46:15.600 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:15 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:15.601 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:15 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:15.601 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:15 smithi053 ceph-mon[24889]: Reconfiguring node-exporter.smithi122 (dependencies changed)... 2024-04-02T04:46:15.601 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:15 smithi053 ceph-mon[24889]: Reconfiguring daemon node-exporter.smithi122 on smithi122 2024-04-02T04:46:15.601 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:15 smithi053 ceph-mon[24889]: pgmap v47: 97 pgs: 97 active+clean; 327 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.0 MiB/s wr, 787 op/s 2024-04-02T04:46:15.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:15 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:15.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:15 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:15.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:15 smithi122 ceph-mon[27145]: Reconfiguring node-exporter.smithi122 (dependencies changed)... 2024-04-02T04:46:15.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:15 smithi122 ceph-mon[27145]: Reconfiguring daemon node-exporter.smithi122 on smithi122 2024-04-02T04:46:15.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:15 smithi122 ceph-mon[27145]: pgmap v47: 97 pgs: 97 active+clean; 327 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.0 MiB/s wr, 787 op/s 2024-04-02T04:46:16.313 INFO:teuthology.orchestra.run.smithi053.stdout: 5 110299 12.29 MB/sec execute 86 sec latency 35.746 ms 2024-04-02T04:46:17.313 INFO:teuthology.orchestra.run.smithi053.stdout: 5 110577 12.23 MB/sec execute 87 sec latency 47.296 ms 2024-04-02T04:46:17.644 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:17 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:17.644 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:17 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:17.644 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:17 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-02T04:46:17.644 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:17 smithi053 ceph-mon[24889]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-02T04:46:17.644 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:17 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi053.front.sepia.ceph.com:9093"}]: dispatch 2024-04-02T04:46:17.644 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:17 smithi053 ceph-mon[24889]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi053.front.sepia.ceph.com:9093"}]: dispatch 2024-04-02T04:46:17.644 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:17 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:17.644 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:17 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-02T04:46:17.645 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:17 smithi053 ceph-mon[24889]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-02T04:46:17.645 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:17 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi053.front.sepia.ceph.com:3000"}]: dispatch 2024-04-02T04:46:17.645 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:17 smithi053 ceph-mon[24889]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi053.front.sepia.ceph.com:3000"}]: dispatch 2024-04-02T04:46:17.645 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:17 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:17.645 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:17 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-02T04:46:17.645 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:17 smithi053 ceph-mon[24889]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-02T04:46:17.645 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:17 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi053.front.sepia.ceph.com:9095"}]: dispatch 2024-04-02T04:46:17.645 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:17 smithi053 ceph-mon[24889]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi053.front.sepia.ceph.com:9095"}]: dispatch 2024-04-02T04:46:17.645 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:17 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:17.645 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:17 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:46:17.645 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:17 smithi053 ceph-mon[24889]: Upgrade: Need to upgrade myself (mgr.smithi122.cpwlyk) 2024-04-02T04:46:17.645 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:17 smithi053 ceph-mon[24889]: Upgrade: Need to upgrade myself (mgr.smithi122.cpwlyk) 2024-04-02T04:46:17.645 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:17 smithi053 ceph-mon[24889]: pgmap v48: 97 pgs: 97 active+clean; 327 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.5 MiB/s wr, 582 op/s 2024-04-02T04:46:17.645 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:17 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:46:17.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:17 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:17.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:17 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:17.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:17 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-02T04:46:17.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:17 smithi122 ceph-mon[27145]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-02T04:46:17.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:17 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi053.front.sepia.ceph.com:9093"}]: dispatch 2024-04-02T04:46:17.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:17 smithi122 ceph-mon[27145]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi053.front.sepia.ceph.com:9093"}]: dispatch 2024-04-02T04:46:17.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:17 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:17.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:17 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-02T04:46:17.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:17 smithi122 ceph-mon[27145]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-02T04:46:17.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:17 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi053.front.sepia.ceph.com:3000"}]: dispatch 2024-04-02T04:46:17.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:17 smithi122 ceph-mon[27145]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi053.front.sepia.ceph.com:3000"}]: dispatch 2024-04-02T04:46:17.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:17 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:17.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:17 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-02T04:46:17.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:17 smithi122 ceph-mon[27145]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-02T04:46:17.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:17 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi053.front.sepia.ceph.com:9095"}]: dispatch 2024-04-02T04:46:17.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:17 smithi122 ceph-mon[27145]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi053.front.sepia.ceph.com:9095"}]: dispatch 2024-04-02T04:46:17.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:17 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:17.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:17 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:46:17.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:17 smithi122 ceph-mon[27145]: Upgrade: Need to upgrade myself (mgr.smithi122.cpwlyk) 2024-04-02T04:46:17.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:17 smithi122 ceph-mon[27145]: Upgrade: Need to upgrade myself (mgr.smithi122.cpwlyk) 2024-04-02T04:46:17.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:17 smithi122 ceph-mon[27145]: pgmap v48: 97 pgs: 97 active+clean; 327 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.5 MiB/s wr, 582 op/s 2024-04-02T04:46:17.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:17 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:46:18.313 INFO:teuthology.orchestra.run.smithi053.stdout: 5 110899 12.15 MB/sec execute 88 sec latency 56.309 ms 2024-04-02T04:46:18.924 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:18 smithi053 ceph-mon[24889]: mgrmap e27: smithi122.cpwlyk(active, since 91s), standbys: smithi053.legdmv 2024-04-02T04:46:19.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:18 smithi122 ceph-mon[27145]: mgrmap e27: smithi122.cpwlyk(active, since 91s), standbys: smithi053.legdmv 2024-04-02T04:46:19.313 INFO:teuthology.orchestra.run.smithi053.stdout: 5 111238 12.08 MB/sec execute 89 sec latency 40.472 ms 2024-04-02T04:46:19.957 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:19 smithi053 ceph-mon[24889]: pgmap v49: 97 pgs: 97 active+clean; 333 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.7 MiB/s wr, 853 op/s 2024-04-02T04:46:20.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:19 smithi122 ceph-mon[27145]: pgmap v49: 97 pgs: 97 active+clean; 333 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.7 MiB/s wr, 853 op/s 2024-04-02T04:46:20.313 INFO:teuthology.orchestra.run.smithi053.stdout: 5 111637 12.07 MB/sec execute 90 sec latency 43.114 ms 2024-04-02T04:46:21.313 INFO:teuthology.orchestra.run.smithi053.stdout: 5 112113 12.04 MB/sec execute 91 sec latency 44.759 ms 2024-04-02T04:46:21.590 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:21 smithi053 ceph-mon[24889]: Upgrade: Updating mgr.smithi053.legdmv 2024-04-02T04:46:21.590 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:21 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:21.590 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:21 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi053.legdmv", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-02T04:46:21.590 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:21 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi053.legdmv", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-02T04:46:21.590 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:21 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-02T04:46:21.590 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:21 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:46:21.590 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:21 smithi053 ceph-mon[24889]: Deploying daemon mgr.smithi053.legdmv on smithi053 2024-04-02T04:46:21.590 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:21 smithi053 ceph-mon[24889]: pgmap v50: 97 pgs: 97 active+clean; 333 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 6.2 MiB/s wr, 584 op/s 2024-04-02T04:46:21.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:21 smithi122 ceph-mon[27145]: Upgrade: Updating mgr.smithi053.legdmv 2024-04-02T04:46:21.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:21 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:21.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:21 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi053.legdmv", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-02T04:46:21.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:21 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi053.legdmv", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-02T04:46:21.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:21 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-02T04:46:21.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:21 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:46:21.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:21 smithi122 ceph-mon[27145]: Deploying daemon mgr.smithi053.legdmv on smithi053 2024-04-02T04:46:21.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:21 smithi122 ceph-mon[27145]: pgmap v50: 97 pgs: 97 active+clean; 333 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 6.2 MiB/s wr, 584 op/s 2024-04-02T04:46:22.313 INFO:teuthology.orchestra.run.smithi053.stdout: 5 112486 12.01 MB/sec execute 92 sec latency 49.204 ms 2024-04-02T04:46:23.314 INFO:teuthology.orchestra.run.smithi053.stdout: 5 113055 12.11 MB/sec execute 93 sec latency 28.340 ms 2024-04-02T04:46:24.165 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:24 smithi053 ceph-mon[24889]: pgmap v51: 97 pgs: 97 active+clean; 333 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 6.2 MiB/s wr, 584 op/s 2024-04-02T04:46:24.314 INFO:teuthology.orchestra.run.smithi053.stdout: 5 113473 12.13 MB/sec execute 94 sec latency 35.764 ms 2024-04-02T04:46:24.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:24 smithi122 ceph-mon[27145]: pgmap v51: 97 pgs: 97 active+clean; 333 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 6.2 MiB/s wr, 584 op/s 2024-04-02T04:46:25.314 INFO:teuthology.orchestra.run.smithi053.stdout: 5 113841 12.13 MB/sec execute 95 sec latency 45.210 ms 2024-04-02T04:46:26.187 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:26 smithi053 ceph-mon[24889]: pgmap v52: 97 pgs: 97 active+clean; 328 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.5 MiB/s wr, 838 op/s 2024-04-02T04:46:26.314 INFO:teuthology.orchestra.run.smithi053.stdout: 5 114172 12.08 MB/sec execute 96 sec latency 33.563 ms 2024-04-02T04:46:26.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:26 smithi122 ceph-mon[27145]: pgmap v52: 97 pgs: 97 active+clean; 328 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.5 MiB/s wr, 838 op/s 2024-04-02T04:46:27.315 INFO:teuthology.orchestra.run.smithi053.stdout: 5 114456 12.01 MB/sec execute 97 sec latency 65.096 ms 2024-04-02T04:46:27.970 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:27 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:27.971 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:27 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:27.971 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:27 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:46:27.971 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:27 smithi053 ceph-mon[24889]: pgmap v53: 97 pgs: 97 active+clean; 328 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.5 MiB/s wr, 523 op/s 2024-04-02T04:46:27.971 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:27 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:27.971 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:27 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:28.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:27 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:28.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:27 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:28.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:27 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:46:28.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:27 smithi122 ceph-mon[27145]: pgmap v53: 97 pgs: 97 active+clean; 328 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.5 MiB/s wr, 523 op/s 2024-04-02T04:46:28.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:27 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:28.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:27 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:28.315 INFO:teuthology.orchestra.run.smithi053.stdout: 5 114846 11.98 MB/sec execute 98 sec latency 40.129 ms 2024-04-02T04:46:29.315 INFO:teuthology.orchestra.run.smithi053.stdout: 5 115250 11.95 MB/sec execute 99 sec latency 46.106 ms 2024-04-02T04:46:30.248 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:30 smithi053 ceph-mon[24889]: pgmap v54: 97 pgs: 97 active+clean; 331 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.0 MiB/s wr, 753 op/s 2024-04-02T04:46:30.281 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:30 smithi122 ceph-mon[27145]: pgmap v54: 97 pgs: 97 active+clean; 331 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.0 MiB/s wr, 753 op/s 2024-04-02T04:46:30.315 INFO:teuthology.orchestra.run.smithi053.stdout: 5 115810 11.94 MB/sec execute 100 sec latency 42.033 ms 2024-04-02T04:46:31.315 INFO:teuthology.orchestra.run.smithi053.stdout: 5 116342 12.02 MB/sec execute 101 sec latency 68.460 ms 2024-04-02T04:46:31.764 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:31 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:31.765 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:31 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:31.765 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:31 smithi053 ceph-mon[24889]: pgmap v55: 97 pgs: 97 active+clean; 331 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.8 MiB/s wr, 482 op/s 2024-04-02T04:46:31.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:31 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:31.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:31 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:31.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:31 smithi122 ceph-mon[27145]: pgmap v55: 97 pgs: 97 active+clean; 331 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.8 MiB/s wr, 482 op/s 2024-04-02T04:46:32.315 INFO:teuthology.orchestra.run.smithi053.stdout: 5 116831 12.06 MB/sec execute 102 sec latency 54.595 ms 2024-04-02T04:46:32.720 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:32 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:46:33.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:32 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:46:33.315 INFO:teuthology.orchestra.run.smithi053.stdout: 5 117253 12.08 MB/sec execute 103 sec latency 57.175 ms 2024-04-02T04:46:34.095 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:34 smithi053 ceph-mon[24889]: pgmap v56: 97 pgs: 97 active+clean; 331 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.8 MiB/s wr, 482 op/s 2024-04-02T04:46:34.315 INFO:teuthology.orchestra.run.smithi053.stdout: 5 117588 12.03 MB/sec execute 104 sec latency 27.922 ms 2024-04-02T04:46:34.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:34 smithi122 ceph-mon[27145]: pgmap v56: 97 pgs: 97 active+clean; 331 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.8 MiB/s wr, 482 op/s 2024-04-02T04:46:35.315 INFO:teuthology.orchestra.run.smithi053.stdout: 5 117976 12.01 MB/sec execute 105 sec latency 44.042 ms 2024-04-02T04:46:36.094 INFO:teuthology.orchestra.run.smithi053.stdout:true 2024-04-02T04:46:36.114 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:36 smithi053 ceph-mon[24889]: pgmap v57: 97 pgs: 97 active+clean; 335 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.7 MiB/s wr, 755 op/s 2024-04-02T04:46:36.315 INFO:teuthology.orchestra.run.smithi053.stdout: 5 118288 11.93 MB/sec execute 106 sec latency 80.811 ms 2024-04-02T04:46:36.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:36 smithi122 ceph-mon[27145]: pgmap v57: 97 pgs: 97 active+clean; 335 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.7 MiB/s wr, 755 op/s 2024-04-02T04:46:37.119 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:37 smithi053 ceph-mon[24889]: from='client.24551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:46:37.119 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:37 smithi053 ceph-mon[24889]: from='mgr.? 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi053.legdmv/crt"}]: dispatch 2024-04-02T04:46:37.120 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:37 smithi053 ceph-mon[24889]: Standby manager daemon smithi053.legdmv restarted 2024-04-02T04:46:37.120 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:37 smithi053 ceph-mon[24889]: Standby manager daemon smithi053.legdmv started 2024-04-02T04:46:37.120 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:37 smithi053 ceph-mon[24889]: from='mgr.? 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-02T04:46:37.120 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:37 smithi053 ceph-mon[24889]: from='mgr.? 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi053.legdmv/key"}]: dispatch 2024-04-02T04:46:37.120 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:37 smithi053 ceph-mon[24889]: from='mgr.? 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-02T04:46:37.150 INFO:teuthology.orchestra.run.smithi053.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-02T04:46:37.150 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager.smithi053 smithi053 *:9093,9094 starting - - - - 2024-04-02T04:46:37.150 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi053 smithi053 running (10m) 95s ago 10m 6409k - 17.2.0 e1d6a67b021e a5c436bbd36d 2024-04-02T04:46:37.151 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi122 smithi122 running (9m) 6s ago 9m 6405k - 17.2.0 e1d6a67b021e b7074474cd46 2024-04-02T04:46:37.151 INFO:teuthology.orchestra.run.smithi053.stdout:grafana.smithi053 smithi053 *:3000 starting - - - - 2024-04-02T04:46:37.151 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi053.nfrfot smithi053 running (4m) 95s ago 4m 35.8M - 17.2.0 e1d6a67b021e c94e9e643514 2024-04-02T04:46:37.151 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi122.mwoyoh smithi122 running (4m) 6s ago 4m 14.7M - 17.2.0 e1d6a67b021e 73376d9ee4b8 2024-04-02T04:46:37.151 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi053.legdmv smithi053 *:8443,9283,8765 starting - - - - 2024-04-02T04:46:37.151 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi122.cpwlyk smithi122 *:8443,9283 running (2m) 6s ago 9m 503M - 19.0.0-1979-g6cec793d 78f830190839 3a534988cf3b 2024-04-02T04:46:37.151 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi053 smithi053 running (11m) 95s ago 11m 51.9M 2048M 17.2.0 e1d6a67b021e 665e4412f6da 2024-04-02T04:46:37.151 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi122 smithi122 running (9m) 6s ago 9m 41.7M 2048M 17.2.0 e1d6a67b021e a2df07e0883d 2024-04-02T04:46:37.151 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.0.0.smithi053.mhkdcb smithi053 *:2049 running (4m) 95s ago 3m 250M - 4.0 e1d6a67b021e 07e5da7bd2c5 2024-04-02T04:46:37.151 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.1.0.smithi122.mauvsb smithi122 *:2049 running (3m) 6s ago 3m 79.2M - 4.0 e1d6a67b021e 4f866685005e 2024-04-02T04:46:37.151 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi053 smithi053 *:9100 starting - - - - 2024-04-02T04:46:37.151 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi122 smithi122 *:9100 running (21s) 6s ago 9m 7926k - 1.3.1 1dbe0e931976 475d439c7980 2024-04-02T04:46:37.151 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 smithi053 running (8m) 95s ago 8m 299M 1937M 17.2.0 e1d6a67b021e 906cb1fd8042 2024-04-02T04:46:37.151 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 smithi053 running (8m) 95s ago 8m 238M 1937M 17.2.0 e1d6a67b021e ac1ad6266e63 2024-04-02T04:46:37.151 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 smithi053 running (7m) 95s ago 7m 208M 1937M 17.2.0 e1d6a67b021e 86964c8a94ff 2024-04-02T04:46:37.152 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 smithi053 running (7m) 95s ago 7m 252M 1937M 17.2.0 e1d6a67b021e 7e8b9e98ae2b 2024-04-02T04:46:37.152 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 smithi122 running (6m) 6s ago 6m 388M 2704M 17.2.0 e1d6a67b021e 46076afcd7bc 2024-04-02T04:46:37.152 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 smithi122 running (6m) 6s ago 6m 427M 2704M 17.2.0 e1d6a67b021e e687a2f6da4d 2024-04-02T04:46:37.152 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 smithi122 running (6m) 6s ago 6m 359M 2704M 17.2.0 e1d6a67b021e 9e371df9a839 2024-04-02T04:46:37.152 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 smithi122 running (5m) 6s ago 5m 493M 2704M 17.2.0 e1d6a67b021e 9b3cc398254d 2024-04-02T04:46:37.152 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus.smithi053 smithi053 *:9095 starting - - - - 2024-04-02T04:46:37.315 INFO:teuthology.orchestra.run.smithi053.stdout: 5 118619 11.91 MB/sec execute 107 sec latency 54.103 ms 2024-04-02T04:46:37.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:37 smithi122 ceph-mon[27145]: from='client.24551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:46:37.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:37 smithi122 ceph-mon[27145]: from='mgr.? 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi053.legdmv/crt"}]: dispatch 2024-04-02T04:46:37.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:37 smithi122 ceph-mon[27145]: Standby manager daemon smithi053.legdmv restarted 2024-04-02T04:46:37.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:37 smithi122 ceph-mon[27145]: Standby manager daemon smithi053.legdmv started 2024-04-02T04:46:37.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:37 smithi122 ceph-mon[27145]: from='mgr.? 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-02T04:46:37.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:37 smithi122 ceph-mon[27145]: from='mgr.? 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi053.legdmv/key"}]: dispatch 2024-04-02T04:46:37.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:37 smithi122 ceph-mon[27145]: from='mgr.? 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-02T04:46:37.740 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:46:37.740 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": { 2024-04-02T04:46:37.740 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:46:37.740 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:46:37.740 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": { 2024-04-02T04:46:37.740 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:46:37.740 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:46:37.740 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": { 2024-04-02T04:46:37.741 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-02T04:46:37.741 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:46:37.741 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": { 2024-04-02T04:46:37.741 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:46:37.741 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:46:37.741 INFO:teuthology.orchestra.run.smithi053.stdout: "overall": { 2024-04-02T04:46:37.741 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-04-02T04:46:37.741 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:46:37.741 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-02T04:46:37.741 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:46:38.130 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:38 smithi053 ceph-mon[24889]: from='client.24555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:46:38.131 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:38 smithi053 ceph-mon[24889]: pgmap v58: 97 pgs: 97 active+clean; 335 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.3 MiB/s wr, 502 op/s 2024-04-02T04:46:38.131 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:38 smithi053 ceph-mon[24889]: mgrmap e28: smithi122.cpwlyk(active, since 111s), standbys: smithi053.legdmv 2024-04-02T04:46:38.131 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:38 smithi053 ceph-mon[24889]: from='client.14768 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:46:38.131 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:38 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/2070420844' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:46:38.278 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:46:38.279 INFO:teuthology.orchestra.run.smithi053.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae", 2024-04-02T04:46:38.279 INFO:teuthology.orchestra.run.smithi053.stdout: "in_progress": true, 2024-04-02T04:46:38.279 INFO:teuthology.orchestra.run.smithi053.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-02T04:46:38.279 INFO:teuthology.orchestra.run.smithi053.stdout: "services_complete": [], 2024-04-02T04:46:38.279 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": "", 2024-04-02T04:46:38.279 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "Currently upgrading mgr daemons", 2024-04-02T04:46:38.279 INFO:teuthology.orchestra.run.smithi053.stdout: "is_paused": false 2024-04-02T04:46:38.279 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:46:38.315 INFO:teuthology.orchestra.run.smithi053.stdout: 5 119209 11.92 MB/sec execute 108 sec latency 39.107 ms 2024-04-02T04:46:38.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:38 smithi122 ceph-mon[27145]: from='client.24555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:46:38.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:38 smithi122 ceph-mon[27145]: pgmap v58: 97 pgs: 97 active+clean; 335 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.3 MiB/s wr, 502 op/s 2024-04-02T04:46:38.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:38 smithi122 ceph-mon[27145]: mgrmap e28: smithi122.cpwlyk(active, since 111s), standbys: smithi053.legdmv 2024-04-02T04:46:38.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:38 smithi122 ceph-mon[27145]: from='client.14768 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:46:38.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:38 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/2070420844' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:46:38.979 INFO:teuthology.orchestra.run.smithi053.stdout:HEALTH_OK 2024-04-02T04:46:39.137 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:39 smithi053 ceph-mon[24889]: from='client.14776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:46:39.137 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:39 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/4283520711' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:46:39.315 INFO:teuthology.orchestra.run.smithi053.stdout: 5 119798 11.99 MB/sec execute 109 sec latency 30.645 ms 2024-04-02T04:46:39.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:39 smithi122 ceph-mon[27145]: from='client.14776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:46:39.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:39 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/4283520711' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:46:40.315 INFO:teuthology.orchestra.run.smithi053.stdout: 5 120329 12.04 MB/sec execute 110 sec latency 33.594 ms 2024-04-02T04:46:40.394 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:40 smithi053 ceph-mon[24889]: pgmap v59: 97 pgs: 97 active+clean; 328 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 741 op/s 2024-04-02T04:46:40.395 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:40 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:40.395 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:40 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:40.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:40 smithi122 ceph-mon[27145]: pgmap v59: 97 pgs: 97 active+clean; 328 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 741 op/s 2024-04-02T04:46:40.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:40 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:40.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:40 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:41.317 INFO:teuthology.orchestra.run.smithi053.stdout: 5 120840 12.08 MB/sec execute 111 sec latency 23.217 ms 2024-04-02T04:46:41.920 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:41 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:41.920 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:41 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:41.920 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:41 smithi053 ceph-mon[24889]: pgmap v60: 97 pgs: 97 active+clean; 328 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.9 MiB/s wr, 511 op/s 2024-04-02T04:46:42.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:41 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:42.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:41 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:42.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:41 smithi122 ceph-mon[27145]: pgmap v60: 97 pgs: 97 active+clean; 328 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.9 MiB/s wr, 511 op/s 2024-04-02T04:46:42.316 INFO:teuthology.orchestra.run.smithi053.stdout: 5 121313 12.09 MB/sec execute 112 sec latency 25.802 ms 2024-04-02T04:46:43.316 INFO:teuthology.orchestra.run.smithi053.stdout: 5 121696 12.03 MB/sec execute 113 sec latency 37.170 ms 2024-04-02T04:46:44.194 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:44 smithi053 ceph-mon[24889]: pgmap v61: 97 pgs: 97 active+clean; 328 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.9 MiB/s wr, 511 op/s 2024-04-02T04:46:44.316 INFO:teuthology.orchestra.run.smithi053.stdout: 5 122063 12.01 MB/sec execute 114 sec latency 41.916 ms 2024-04-02T04:46:44.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:44 smithi122 ceph-mon[27145]: pgmap v61: 97 pgs: 97 active+clean; 328 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.9 MiB/s wr, 511 op/s 2024-04-02T04:46:45.316 INFO:teuthology.orchestra.run.smithi053.stdout: 5 122571 11.99 MB/sec execute 115 sec latency 53.781 ms 2024-04-02T04:46:46.021 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:46 smithi053 ceph-mon[24889]: pgmap v62: 97 pgs: 97 active+clean; 336 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 8.4 MiB/s wr, 817 op/s 2024-04-02T04:46:46.315 INFO:teuthology.orchestra.run.smithi053.stdout: 5 122987 11.96 MB/sec execute 116 sec latency 48.391 ms 2024-04-02T04:46:46.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:46 smithi122 ceph-mon[27145]: pgmap v62: 97 pgs: 97 active+clean; 336 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 8.4 MiB/s wr, 817 op/s 2024-04-02T04:46:47.230 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:47 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:46:47.316 INFO:teuthology.orchestra.run.smithi053.stdout: 5 123421 12.02 MB/sec execute 117 sec latency 60.653 ms 2024-04-02T04:46:47.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:47 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:46:48.255 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:48 smithi053 ceph-mon[24889]: pgmap v63: 97 pgs: 97 active+clean; 336 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.5 MiB/s wr, 544 op/s 2024-04-02T04:46:48.316 INFO:teuthology.orchestra.run.smithi053.stdout: 5 123841 12.01 MB/sec execute 118 sec latency 47.909 ms 2024-04-02T04:46:48.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:48 smithi122 ceph-mon[27145]: pgmap v63: 97 pgs: 97 active+clean; 336 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.5 MiB/s wr, 544 op/s 2024-04-02T04:46:49.316 INFO:teuthology.orchestra.run.smithi053.stdout: 5 124226 12.03 MB/sec execute 119 sec latency 41.565 ms 2024-04-02T04:46:50.020 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:49 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:50.021 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:49 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:50.021 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:49 smithi053 ceph-mon[24889]: pgmap v64: 97 pgs: 97 active+clean; 333 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.2 MiB/s wr, 808 op/s 2024-04-02T04:46:50.021 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:49 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:46:50.021 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:49 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:46:50.021 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:49 smithi053 ceph-mon[24889]: pgmap v65: 97 pgs: 97 active+clean; 333 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 7.3 MiB/s wr, 682 op/s 2024-04-02T04:46:50.021 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:49 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:50.021 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:49 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:50.021 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:49 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:46:50.021 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:49 smithi053 ceph-mon[24889]: Upgrade: Need to upgrade myself (mgr.smithi122.cpwlyk) 2024-04-02T04:46:50.021 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:49 smithi053 ceph-mon[24889]: Upgrade: Need to upgrade myself (mgr.smithi122.cpwlyk) 2024-04-02T04:46:50.021 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:49 smithi053 ceph-mon[24889]: Failing over to other MGR 2024-04-02T04:46:50.021 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:49 smithi053 ceph-mon[24889]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "mgr fail", "who": "smithi122.cpwlyk"}]: dispatch 2024-04-02T04:46:50.021 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:49 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "mgr fail", "who": "smithi122.cpwlyk"}]: dispatch 2024-04-02T04:46:50.022 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:49 smithi053 ceph-mon[24889]: osdmap e71: 8 total, 8 up, 8 in 2024-04-02T04:46:50.022 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:49 smithi053 ceph-mon[24889]: Standby manager daemon smithi122.cpwlyk started 2024-04-02T04:46:50.223 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:49 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:50.224 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:49 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:50.224 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:49 smithi122 ceph-mon[27145]: pgmap v64: 97 pgs: 97 active+clean; 333 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.2 MiB/s wr, 808 op/s 2024-04-02T04:46:50.224 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:49 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:46:50.224 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:49 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:46:50.224 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:49 smithi122 ceph-mon[27145]: pgmap v65: 97 pgs: 97 active+clean; 333 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 7.3 MiB/s wr, 682 op/s 2024-04-02T04:46:50.224 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:49 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:50.224 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:49 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' 2024-04-02T04:46:50.224 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:49 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:46:50.224 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:49 smithi122 ceph-mon[27145]: Upgrade: Need to upgrade myself (mgr.smithi122.cpwlyk) 2024-04-02T04:46:50.224 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:49 smithi122 ceph-mon[27145]: Upgrade: Need to upgrade myself (mgr.smithi122.cpwlyk) 2024-04-02T04:46:50.224 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:49 smithi122 ceph-mon[27145]: Failing over to other MGR 2024-04-02T04:46:50.224 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:49 smithi122 ceph-mon[27145]: from='mgr.24457 172.21.15.122:0/2485496490' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "mgr fail", "who": "smithi122.cpwlyk"}]: dispatch 2024-04-02T04:46:50.224 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:49 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "mgr fail", "who": "smithi122.cpwlyk"}]: dispatch 2024-04-02T04:46:50.224 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:49 smithi122 ceph-mon[27145]: osdmap e71: 8 total, 8 up, 8 in 2024-04-02T04:46:50.225 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:49 smithi122 ceph-mon[27145]: Standby manager daemon smithi122.cpwlyk started 2024-04-02T04:46:50.316 INFO:teuthology.orchestra.run.smithi053.stdout: 5 124499 12.00 MB/sec execute 120 sec latency 38.779 ms 2024-04-02T04:46:51.037 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:50 smithi053 ceph-mon[24889]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-02T04:46:51.037 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:50 smithi053 ceph-mon[24889]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' cmd='[{"prefix": "mgr fail", "who": "smithi122.cpwlyk"}]': finished 2024-04-02T04:46:51.037 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:50 smithi053 ceph-mon[24889]: mgrmap e29: smithi053.legdmv(active, starting, since 0.854988s), standbys: smithi122.cpwlyk 2024-04-02T04:46:51.037 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:50 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi053"}]: dispatch 2024-04-02T04:46:51.037 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:50 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-02T04:46:51.037 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:50 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi053.nfrfot"}]: dispatch 2024-04-02T04:46:51.037 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:50 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi122.mwoyoh"}]: dispatch 2024-04-02T04:46:51.038 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:50 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mgr metadata", "who": "smithi053.legdmv", "id": "smithi053.legdmv"}]: dispatch 2024-04-02T04:46:51.038 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:50 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mgr metadata", "who": "smithi122.cpwlyk", "id": "smithi122.cpwlyk"}]: dispatch 2024-04-02T04:46:51.038 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:50 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:46:51.038 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:50 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:46:51.038 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:50 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:46:51.038 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:50 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:46:51.038 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:50 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:46:51.038 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:50 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:46:51.038 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:50 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:46:51.038 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:50 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:46:51.038 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:50 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-02T04:46:51.038 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:50 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-02T04:46:51.038 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:50 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-02T04:46:51.039 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:50 smithi053 ceph-mon[24889]: Manager daemon smithi053.legdmv is now available 2024-04-02T04:46:51.039 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:50 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:46:51.039 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:50 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:46:51.039 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:50 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:46:51.039 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:50 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:46:51.039 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:50 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/trash_purge_schedule"}]: dispatch 2024-04-02T04:46:51.316 INFO:teuthology.orchestra.run.smithi053.stdout: 5 124939 11.98 MB/sec execute 121 sec latency 32.837 ms 2024-04-02T04:46:51.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:50 smithi122 ceph-mon[27145]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-02T04:46:51.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:50 smithi122 ceph-mon[27145]: from='mgr.24457 ' entity='mgr.smithi122.cpwlyk' cmd='[{"prefix": "mgr fail", "who": "smithi122.cpwlyk"}]': finished 2024-04-02T04:46:51.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:50 smithi122 ceph-mon[27145]: mgrmap e29: smithi053.legdmv(active, starting, since 0.854988s), standbys: smithi122.cpwlyk 2024-04-02T04:46:51.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:50 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi053"}]: dispatch 2024-04-02T04:46:51.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:50 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-02T04:46:51.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:50 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi053.nfrfot"}]: dispatch 2024-04-02T04:46:51.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:50 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi122.mwoyoh"}]: dispatch 2024-04-02T04:46:51.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:50 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mgr metadata", "who": "smithi053.legdmv", "id": "smithi053.legdmv"}]: dispatch 2024-04-02T04:46:51.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:50 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mgr metadata", "who": "smithi122.cpwlyk", "id": "smithi122.cpwlyk"}]: dispatch 2024-04-02T04:46:51.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:50 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:46:51.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:50 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:46:51.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:50 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:46:51.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:50 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:46:51.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:50 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:46:51.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:50 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:46:51.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:50 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:46:51.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:50 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:46:51.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:50 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-02T04:46:51.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:50 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-02T04:46:51.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:50 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-02T04:46:51.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:50 smithi122 ceph-mon[27145]: Manager daemon smithi053.legdmv is now available 2024-04-02T04:46:51.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:50 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:46:51.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:50 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:46:51.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:50 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:46:51.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:50 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:46:51.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:50 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/trash_purge_schedule"}]: dispatch 2024-04-02T04:46:52.040 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:51 smithi053 ceph-mon[24889]: mgrmap e30: smithi053.legdmv(active, since 1.86307s), standbys: smithi122.cpwlyk 2024-04-02T04:46:52.041 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:51 smithi053 ceph-mon[24889]: pgmap v3: 97 pgs: 97 active+clean; 339 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-04-02T04:46:52.283 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:51 smithi122 ceph-mon[27145]: mgrmap e30: smithi053.legdmv(active, since 1.86307s), standbys: smithi122.cpwlyk 2024-04-02T04:46:52.283 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:51 smithi122 ceph-mon[27145]: pgmap v3: 97 pgs: 97 active+clean; 339 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-04-02T04:46:52.316 INFO:teuthology.orchestra.run.smithi053.stdout: 5 125388 11.94 MB/sec execute 122 sec latency 33.536 ms 2024-04-02T04:46:53.048 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:52 smithi053 ceph-mon[24889]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-02T04:46:53.049 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:53 smithi053 ceph-mon[24889]: Cluster is now healthy 2024-04-02T04:46:53.049 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:53 smithi053 ceph-mon[24889]: [02/Apr/2024:04:46:52] ENGINE Bus STARTING 2024-04-02T04:46:53.049 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:53 smithi053 ceph-mon[24889]: [02/Apr/2024:04:46:52] ENGINE Serving on https://172.21.15.53:7150 2024-04-02T04:46:53.049 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:53 smithi053 ceph-mon[24889]: [02/Apr/2024:04:46:52] ENGINE Serving on http://172.21.15.53:8765 2024-04-02T04:46:53.049 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:53 smithi053 ceph-mon[24889]: [02/Apr/2024:04:46:52] ENGINE Bus STARTED 2024-04-02T04:46:53.231 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:52 smithi122 ceph-mon[27145]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-02T04:46:53.232 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:52 smithi122 ceph-mon[27145]: Cluster is now healthy 2024-04-02T04:46:53.232 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:52 smithi122 ceph-mon[27145]: [02/Apr/2024:04:46:52] ENGINE Bus STARTING 2024-04-02T04:46:53.232 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:52 smithi122 ceph-mon[27145]: [02/Apr/2024:04:46:52] ENGINE Serving on https://172.21.15.53:7150 2024-04-02T04:46:53.232 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:52 smithi122 ceph-mon[27145]: [02/Apr/2024:04:46:52] ENGINE Serving on http://172.21.15.53:8765 2024-04-02T04:46:53.232 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:52 smithi122 ceph-mon[27145]: [02/Apr/2024:04:46:52] ENGINE Bus STARTED 2024-04-02T04:46:53.316 INFO:teuthology.orchestra.run.smithi053.stdout: 5 125729 11.91 MB/sec execute 123 sec latency 69.595 ms 2024-04-02T04:46:54.063 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:53 smithi053 ceph-mon[24889]: mgrmap e31: smithi053.legdmv(active, since 3s), standbys: smithi122.cpwlyk 2024-04-02T04:46:54.277 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:54 smithi122 ceph-mon[27145]: mgrmap e31: smithi053.legdmv(active, since 3s), standbys: smithi122.cpwlyk 2024-04-02T04:46:54.316 INFO:teuthology.orchestra.run.smithi053.stdout: 5 126112 11.88 MB/sec execute 124 sec latency 73.153 ms 2024-04-02T04:46:55.073 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:55 smithi053 ceph-mon[24889]: pgmap v5: 97 pgs: 97 active+clean; 339 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-04-02T04:46:55.073 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:55 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:46:55.074 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:55 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:46:55.277 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:55 smithi122 ceph-mon[27145]: pgmap v5: 97 pgs: 97 active+clean; 339 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-04-02T04:46:55.278 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:55 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:46:55.278 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:55 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:46:55.316 INFO:teuthology.orchestra.run.smithi053.stdout: 5 126458 11.84 MB/sec execute 125 sec latency 38.705 ms 2024-04-02T04:46:56.316 INFO:teuthology.orchestra.run.smithi053.stdout: 5 126745 11.86 MB/sec execute 126 sec latency 54.706 ms 2024-04-02T04:46:57.316 INFO:teuthology.orchestra.run.smithi053.stdout: 5 127056 11.85 MB/sec execute 127 sec latency 58.435 ms 2024-04-02T04:46:57.398 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:57 smithi053 ceph-mon[24889]: pgmap v6: 97 pgs: 97 active+clean; 337 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 532 KiB/s wr, 41 op/s 2024-04-02T04:46:57.398 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:57 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:46:57.399 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:57 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:46:57.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:57 smithi122 ceph-mon[27145]: pgmap v6: 97 pgs: 97 active+clean; 337 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 532 KiB/s wr, 41 op/s 2024-04-02T04:46:57.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:57 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:46:57.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:57 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:46:58.316 INFO:teuthology.orchestra.run.smithi053.stdout: 5 127469 11.85 MB/sec execute 128 sec latency 29.601 ms 2024-04-02T04:46:59.316 INFO:teuthology.orchestra.run.smithi053.stdout: 5 127754 11.84 MB/sec execute 129 sec latency 38.046 ms 2024-04-02T04:46:59.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:59 smithi053 ceph-mon[24889]: pgmap v7: 97 pgs: 97 active+clean; 335 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 36 KiB/s rd, 3.5 MiB/s wr, 417 op/s 2024-04-02T04:46:59.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:59 smithi053 ceph-mon[24889]: Standby manager daemon smithi122.cpwlyk restarted 2024-04-02T04:46:59.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:59 smithi053 ceph-mon[24889]: Standby manager daemon smithi122.cpwlyk started 2024-04-02T04:46:59.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:59 smithi053 ceph-mon[24889]: from='mgr.? 172.21.15.122:0/302815187' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.cpwlyk/crt"}]: dispatch 2024-04-02T04:46:59.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:59 smithi053 ceph-mon[24889]: from='mgr.? 172.21.15.122:0/302815187' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-02T04:46:59.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:59 smithi053 ceph-mon[24889]: from='mgr.? 172.21.15.122:0/302815187' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.cpwlyk/key"}]: dispatch 2024-04-02T04:46:59.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:46:59 smithi053 ceph-mon[24889]: from='mgr.? 172.21.15.122:0/302815187' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-02T04:46:59.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:59 smithi122 ceph-mon[27145]: pgmap v7: 97 pgs: 97 active+clean; 335 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 36 KiB/s rd, 3.5 MiB/s wr, 417 op/s 2024-04-02T04:46:59.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:59 smithi122 ceph-mon[27145]: Standby manager daemon smithi122.cpwlyk restarted 2024-04-02T04:46:59.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:59 smithi122 ceph-mon[27145]: Standby manager daemon smithi122.cpwlyk started 2024-04-02T04:46:59.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:59 smithi122 ceph-mon[27145]: from='mgr.? 172.21.15.122:0/302815187' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.cpwlyk/crt"}]: dispatch 2024-04-02T04:46:59.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:59 smithi122 ceph-mon[27145]: from='mgr.? 172.21.15.122:0/302815187' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-02T04:46:59.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:59 smithi122 ceph-mon[27145]: from='mgr.? 172.21.15.122:0/302815187' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.cpwlyk/key"}]: dispatch 2024-04-02T04:46:59.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:46:59 smithi122 ceph-mon[27145]: from='mgr.? 172.21.15.122:0/302815187' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-02T04:47:00.316 INFO:teuthology.orchestra.run.smithi053.stdout: 5 127980 11.80 MB/sec execute 130 sec latency 44.682 ms 2024-04-02T04:47:00.449 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:00 smithi053 ceph-mon[24889]: mgrmap e32: smithi053.legdmv(active, since 10s), standbys: smithi122.cpwlyk 2024-04-02T04:47:00.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:00 smithi122 ceph-mon[27145]: mgrmap e32: smithi053.legdmv(active, since 10s), standbys: smithi122.cpwlyk 2024-04-02T04:47:01.317 INFO:teuthology.orchestra.run.smithi053.stdout: 5 128306 11.75 MB/sec execute 131 sec latency 53.942 ms 2024-04-02T04:47:01.477 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:01 smithi053 ceph-mon[24889]: pgmap v8: 97 pgs: 97 active+clean; 335 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 28 KiB/s rd, 2.7 MiB/s wr, 324 op/s 2024-04-02T04:47:01.477 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:01 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:01.477 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:01 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:01.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:01 smithi122 ceph-mon[27145]: pgmap v8: 97 pgs: 97 active+clean; 335 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 28 KiB/s rd, 2.7 MiB/s wr, 324 op/s 2024-04-02T04:47:01.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:01 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:01.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:01 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:02.317 INFO:teuthology.orchestra.run.smithi053.stdout: 5 128614 11.72 MB/sec execute 132 sec latency 67.535 ms 2024-04-02T04:47:03.318 INFO:teuthology.orchestra.run.smithi053.stdout: 5 128873 11.66 MB/sec execute 133 sec latency 56.552 ms 2024-04-02T04:47:03.509 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:03 smithi053 ceph-mon[24889]: pgmap v9: 97 pgs: 97 active+clean; 343 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 4.7 MiB/s wr, 470 op/s 2024-04-02T04:47:03.509 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:03 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:03.509 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:03 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:03.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:03 smithi122 ceph-mon[27145]: pgmap v9: 97 pgs: 97 active+clean; 343 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 4.7 MiB/s wr, 470 op/s 2024-04-02T04:47:03.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:03 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:03.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:03 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:04.318 INFO:teuthology.orchestra.run.smithi053.stdout: 5 129366 11.67 MB/sec execute 134 sec latency 37.276 ms 2024-04-02T04:47:05.278 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:05 smithi053 ceph-mon[24889]: pgmap v10: 97 pgs: 97 active+clean; 343 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 4.3 MiB/s wr, 430 op/s 2024-04-02T04:47:05.278 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:05 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:05.279 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:05 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:47:05.318 INFO:teuthology.orchestra.run.smithi053.stdout: 5 129924 11.67 MB/sec execute 135 sec latency 50.226 ms 2024-04-02T04:47:05.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:05 smithi122 ceph-mon[27145]: pgmap v10: 97 pgs: 97 active+clean; 343 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 4.3 MiB/s wr, 430 op/s 2024-04-02T04:47:05.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:05 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:05.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:05 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:47:06.318 INFO:teuthology.orchestra.run.smithi053.stdout: 5 130338 11.70 MB/sec execute 136 sec latency 32.031 ms 2024-04-02T04:47:07.308 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:07 smithi053 ceph-mon[24889]: pgmap v11: 97 pgs: 97 active+clean; 343 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 4.3 MiB/s wr, 430 op/s 2024-04-02T04:47:07.317 INFO:teuthology.orchestra.run.smithi053.stdout: 5 130735 11.71 MB/sec execute 137 sec latency 50.292 ms 2024-04-02T04:47:07.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:07 smithi122 ceph-mon[27145]: pgmap v11: 97 pgs: 97 active+clean; 343 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 4.3 MiB/s wr, 430 op/s 2024-04-02T04:47:08.318 INFO:teuthology.orchestra.run.smithi053.stdout: 5 131062 11.69 MB/sec execute 138 sec latency 43.864 ms 2024-04-02T04:47:09.158 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:09 smithi053 ceph-mon[24889]: pgmap v12: 97 pgs: 97 active+clean; 336 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 6.2 MiB/s wr, 660 op/s 2024-04-02T04:47:09.158 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:09 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:09.159 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:09 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:09.159 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:09 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:47:09.159 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:09 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:47:09.159 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:09 smithi053 ceph-mon[24889]: Updating smithi053:/etc/ceph/ceph.conf 2024-04-02T04:47:09.159 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:09 smithi053 ceph-mon[24889]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-02T04:47:09.159 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:09 smithi053 ceph-mon[24889]: Updating smithi053:/var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T04:47:09.159 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:09 smithi053 ceph-mon[24889]: Updating smithi122:/var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T04:47:09.159 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:09 smithi053 ceph-mon[24889]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-04-02T04:47:09.318 INFO:teuthology.orchestra.run.smithi053.stdout: 5 131331 11.68 MB/sec execute 139 sec latency 63.998 ms 2024-04-02T04:47:09.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:09 smithi122 ceph-mon[27145]: pgmap v12: 97 pgs: 97 active+clean; 336 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 6.2 MiB/s wr, 660 op/s 2024-04-02T04:47:09.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:09 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:09.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:09 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:09.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:09 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:47:09.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:09 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:47:09.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:09 smithi122 ceph-mon[27145]: Updating smithi053:/etc/ceph/ceph.conf 2024-04-02T04:47:09.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:09 smithi122 ceph-mon[27145]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-02T04:47:09.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:09 smithi122 ceph-mon[27145]: Updating smithi053:/var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T04:47:09.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:09 smithi122 ceph-mon[27145]: Updating smithi122:/var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T04:47:09.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:09 smithi122 ceph-mon[27145]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-04-02T04:47:09.431 INFO:teuthology.orchestra.run.smithi053.stdout:true 2024-04-02T04:47:10.174 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:10 smithi053 ceph-mon[24889]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-02T04:47:10.174 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:10 smithi053 ceph-mon[24889]: Updating smithi053:/var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T04:47:10.174 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:10 smithi053 ceph-mon[24889]: Updating smithi122:/var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T04:47:10.174 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:10 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:10.174 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:10 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:10.174 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:10 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:10.175 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:10 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:10.175 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:10 smithi053 ceph-mon[24889]: pgmap v13: 97 pgs: 97 active+clean; 336 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.7 MiB/s wr, 462 op/s 2024-04-02T04:47:10.175 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:10 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:10.175 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:10 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:10.175 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:10 smithi053 ceph-mon[24889]: from='client.14810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:47:10.175 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:10 smithi053 ceph-mon[24889]: Reconfiguring prometheus.smithi053 (dependencies changed)... 2024-04-02T04:47:10.318 INFO:teuthology.orchestra.run.smithi053.stdout: 5 131556 11.64 MB/sec execute 140 sec latency 45.730 ms 2024-04-02T04:47:10.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:10 smithi122 ceph-mon[27145]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-02T04:47:10.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:10 smithi122 ceph-mon[27145]: Updating smithi053:/var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T04:47:10.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:10 smithi122 ceph-mon[27145]: Updating smithi122:/var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T04:47:10.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:10 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:10.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:10 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:10.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:10 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:10.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:10 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:10.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:10 smithi122 ceph-mon[27145]: pgmap v13: 97 pgs: 97 active+clean; 336 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.7 MiB/s wr, 462 op/s 2024-04-02T04:47:10.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:10 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:10.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:10 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:10.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:10 smithi122 ceph-mon[27145]: from='client.14810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:47:10.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:10 smithi122 ceph-mon[27145]: Reconfiguring prometheus.smithi053 (dependencies changed)... 2024-04-02T04:47:10.456 INFO:teuthology.orchestra.run.smithi053.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-02T04:47:10.457 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager.smithi053 smithi053 *:9093,9094 running (106s) 9s ago 10m 25.7M - 0.23.0 ba2b418f427c bf50c1cd2b3d 2024-04-02T04:47:10.457 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi053 smithi053 running (10m) 9s ago 10m 6409k - 17.2.0 e1d6a67b021e a5c436bbd36d 2024-04-02T04:47:10.457 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi122 smithi122 running (10m) 15s ago 10m 6405k - 17.2.0 e1d6a67b021e b7074474cd46 2024-04-02T04:47:10.457 INFO:teuthology.orchestra.run.smithi053.stdout:grafana.smithi053 smithi053 *:3000 unknown 9s ago 10m - - 2024-04-02T04:47:10.457 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi053.nfrfot smithi053 running (4m) 9s ago 4m 36.5M - 17.2.0 e1d6a67b021e c94e9e643514 2024-04-02T04:47:10.457 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi122.mwoyoh smithi122 running (4m) 15s ago 4m 15.0M - 17.2.0 e1d6a67b021e 73376d9ee4b8 2024-04-02T04:47:10.457 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi053.legdmv smithi053 *:8443,9283,8765 running (44s) 9s ago 12m 476M - 19.0.0-1979-g6cec793d 78f830190839 c12d516a31bc 2024-04-02T04:47:10.457 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi122.cpwlyk smithi122 *:8443,9283 running (2m) 15s ago 10m 81.1M - 19.0.0-1979-g6cec793d 78f830190839 3a534988cf3b 2024-04-02T04:47:10.457 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi053 smithi053 running (12m) 9s ago 12m 54.2M 2048M 17.2.0 e1d6a67b021e 665e4412f6da 2024-04-02T04:47:10.457 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi122 smithi122 running (9m) 15s ago 9m 43.2M 2048M 17.2.0 e1d6a67b021e a2df07e0883d 2024-04-02T04:47:10.457 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.0.0.smithi053.mhkdcb smithi053 *:2049 running (4m) 9s ago 4m 265M - 4.0 e1d6a67b021e 07e5da7bd2c5 2024-04-02T04:47:10.457 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.1.0.smithi122.mauvsb smithi122 *:2049 running (4m) 15s ago 4m 79.5M - 4.0 e1d6a67b021e 4f866685005e 2024-04-02T04:47:10.457 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi053 smithi053 *:9100 running (79s) 9s ago 10m 22.4M - 1.3.1 1dbe0e931976 773b71971e67 2024-04-02T04:47:10.457 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi122 smithi122 *:9100 running (54s) 15s ago 9m 9852k - 1.3.1 1dbe0e931976 475d439c7980 2024-04-02T04:47:10.458 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 smithi053 running (9m) 9s ago 9m 462M 1937M 17.2.0 e1d6a67b021e 906cb1fd8042 2024-04-02T04:47:10.458 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 smithi053 running (8m) 9s ago 8m 404M 1937M 17.2.0 e1d6a67b021e ac1ad6266e63 2024-04-02T04:47:10.458 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 smithi053 running (8m) 9s ago 8m 315M 1937M 17.2.0 e1d6a67b021e 86964c8a94ff 2024-04-02T04:47:10.458 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 smithi053 running (7m) 9s ago 7m 414M 1937M 17.2.0 e1d6a67b021e 7e8b9e98ae2b 2024-04-02T04:47:10.458 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 smithi122 running (7m) 15s ago 7m 432M 2704M 17.2.0 e1d6a67b021e 46076afcd7bc 2024-04-02T04:47:10.458 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 smithi122 running (7m) 15s ago 7m 461M 2704M 17.2.0 e1d6a67b021e e687a2f6da4d 2024-04-02T04:47:10.458 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 smithi122 running (6m) 15s ago 6m 403M 2704M 17.2.0 e1d6a67b021e 9e371df9a839 2024-04-02T04:47:10.458 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 smithi122 running (6m) 15s ago 6m 523M 2704M 17.2.0 e1d6a67b021e 9b3cc398254d 2024-04-02T04:47:10.458 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus.smithi053 smithi053 *:9095 running (56s) 9s ago 9m 52.0M - 2.33.4 514e6a882f6e beded8bd13af 2024-04-02T04:47:11.048 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:47:11.048 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": { 2024-04-02T04:47:11.048 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:47:11.048 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:47:11.048 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": { 2024-04-02T04:47:11.048 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:47:11.048 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:47:11.048 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": { 2024-04-02T04:47:11.048 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-02T04:47:11.048 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:47:11.048 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": { 2024-04-02T04:47:11.048 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:47:11.048 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:47:11.048 INFO:teuthology.orchestra.run.smithi053.stdout: "overall": { 2024-04-02T04:47:11.049 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-04-02T04:47:11.049 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:47:11.049 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-02T04:47:11.049 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:47:11.179 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:11 smithi053 ceph-mon[24889]: Reconfiguring daemon prometheus.smithi053 on smithi053 2024-04-02T04:47:11.180 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:11 smithi053 ceph-mon[24889]: from='client.14814 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:47:11.180 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:11 smithi053 ceph-mon[24889]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-02T04:47:11.180 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:11 smithi053 ceph-mon[24889]: from='client.14818 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:47:11.180 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:11 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/3983960979' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:47:11.318 INFO:teuthology.orchestra.run.smithi053.stdout: 5 131902 11.61 MB/sec execute 141 sec latency 48.593 ms 2024-04-02T04:47:11.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:11 smithi122 ceph-mon[27145]: Reconfiguring daemon prometheus.smithi053 on smithi053 2024-04-02T04:47:11.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:11 smithi122 ceph-mon[27145]: from='client.14814 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:47:11.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:11 smithi122 ceph-mon[27145]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-02T04:47:11.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:11 smithi122 ceph-mon[27145]: from='client.14818 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:47:11.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:11 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/3983960979' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:47:11.559 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:47:11.560 INFO:teuthology.orchestra.run.smithi053.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae", 2024-04-02T04:47:11.560 INFO:teuthology.orchestra.run.smithi053.stdout: "in_progress": true, 2024-04-02T04:47:11.560 INFO:teuthology.orchestra.run.smithi053.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-02T04:47:11.560 INFO:teuthology.orchestra.run.smithi053.stdout: "services_complete": [ 2024-04-02T04:47:11.560 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr" 2024-04-02T04:47:11.560 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-04-02T04:47:11.560 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": "2/23 daemons upgraded", 2024-04-02T04:47:11.560 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "", 2024-04-02T04:47:11.560 INFO:teuthology.orchestra.run.smithi053.stdout: "is_paused": false 2024-04-02T04:47:11.560 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:47:12.179 INFO:teuthology.orchestra.run.smithi053.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-02T04:47:12.180 INFO:teuthology.orchestra.run.smithi053.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-02T04:47:12.180 INFO:teuthology.orchestra.run.smithi053.stdout: daemon grafana.smithi053 on smithi053 is in unknown state 2024-04-02T04:47:12.200 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:12 smithi053 ceph-mon[24889]: pgmap v14: 97 pgs: 97 active+clean; 340 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.1 MiB/s wr, 658 op/s 2024-04-02T04:47:12.200 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:12 smithi053 ceph-mon[24889]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:47:12.318 INFO:teuthology.orchestra.run.smithi053.stdout: 5 132240 11.58 MB/sec execute 142 sec latency 37.086 ms 2024-04-02T04:47:12.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:12 smithi122 ceph-mon[27145]: pgmap v14: 97 pgs: 97 active+clean; 340 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.1 MiB/s wr, 658 op/s 2024-04-02T04:47:12.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:12 smithi122 ceph-mon[27145]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:47:13.216 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:13 smithi053 ceph-mon[24889]: from='client.? 172.21.15.53:0/1456574846' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:47:13.318 INFO:teuthology.orchestra.run.smithi053.stdout: 5 132703 11.58 MB/sec execute 143 sec latency 34.248 ms 2024-04-02T04:47:13.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:13 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/1456574846' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:47:14.224 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:14 smithi053 ceph-mon[24889]: pgmap v15: 97 pgs: 97 active+clean; 340 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.7 MiB/s wr, 459 op/s 2024-04-02T04:47:14.318 INFO:teuthology.orchestra.run.smithi053.stdout: 5 133223 11.55 MB/sec execute 144 sec latency 42.742 ms 2024-04-02T04:47:14.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:14 smithi122 ceph-mon[27145]: pgmap v15: 97 pgs: 97 active+clean; 340 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.7 MiB/s wr, 459 op/s 2024-04-02T04:47:15.318 INFO:teuthology.orchestra.run.smithi053.stdout: 5 133629 11.54 MB/sec execute 145 sec latency 59.768 ms 2024-04-02T04:47:16.318 INFO:teuthology.orchestra.run.smithi053.stdout: 5 133939 11.56 MB/sec execute 146 sec latency 48.513 ms 2024-04-02T04:47:16.578 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:16 smithi053 ceph-mon[24889]: pgmap v16: 97 pgs: 97 active+clean; 340 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.7 MiB/s wr, 459 op/s 2024-04-02T04:47:16.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:16 smithi122 ceph-mon[27145]: pgmap v16: 97 pgs: 97 active+clean; 340 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.7 MiB/s wr, 459 op/s 2024-04-02T04:47:17.318 INFO:teuthology.orchestra.run.smithi053.stdout: 5 134266 11.56 MB/sec execute 147 sec latency 69.156 ms 2024-04-02T04:47:18.318 INFO:teuthology.orchestra.run.smithi053.stdout: 5 134587 11.55 MB/sec execute 148 sec latency 45.255 ms 2024-04-02T04:47:18.510 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:18 smithi053 ceph-mon[24889]: pgmap v17: 97 pgs: 97 active+clean; 334 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.1 MiB/s wr, 730 op/s 2024-04-02T04:47:18.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:18 smithi122 ceph-mon[27145]: pgmap v17: 97 pgs: 97 active+clean; 334 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.1 MiB/s wr, 730 op/s 2024-04-02T04:47:19.318 INFO:teuthology.orchestra.run.smithi053.stdout: 5 134812 11.53 MB/sec execute 149 sec latency 61.360 ms 2024-04-02T04:47:20.318 INFO:teuthology.orchestra.run.smithi053.stdout: 5 135140 11.51 MB/sec execute 150 sec latency 46.060 ms 2024-04-02T04:47:20.538 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:20 smithi053 ceph-mon[24889]: pgmap v18: 97 pgs: 97 active+clean; 334 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.8 MiB/s wr, 468 op/s 2024-04-02T04:47:20.538 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:20 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:20.539 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:20 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:20.539 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:20 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-02T04:47:20.539 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:20 smithi053 ceph-mon[24889]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-02T04:47:20.539 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:20 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:47:20.539 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:20 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:20.539 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:20 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:47:20.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:20 smithi122 ceph-mon[27145]: pgmap v18: 97 pgs: 97 active+clean; 334 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.8 MiB/s wr, 468 op/s 2024-04-02T04:47:20.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:20 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:20.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:20 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:20.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:20 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-02T04:47:20.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:20 smithi122 ceph-mon[27145]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-02T04:47:20.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:20 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:47:20.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:20 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:20.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:20 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:47:21.318 INFO:teuthology.orchestra.run.smithi053.stdout: 5 135523 11.48 MB/sec execute 151 sec latency 62.723 ms 2024-04-02T04:47:21.555 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:21 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi122.cpwlyk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-02T04:47:21.555 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:21 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-02T04:47:21.556 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:21 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:47:21.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:21 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi122.cpwlyk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-02T04:47:21.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:21 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-02T04:47:21.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:21 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:47:22.318 INFO:teuthology.orchestra.run.smithi053.stdout: 5 135826 11.45 MB/sec execute 152 sec latency 40.191 ms 2024-04-02T04:47:22.572 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:22 smithi053 ceph-mon[24889]: Upgrade: Updating mgr.smithi122.cpwlyk 2024-04-02T04:47:22.572 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:22 smithi053 ceph-mon[24889]: Deploying daemon mgr.smithi122.cpwlyk on smithi122 2024-04-02T04:47:22.572 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:22 smithi053 ceph-mon[24889]: pgmap v19: 97 pgs: 97 active+clean; 341 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 6.7 MiB/s wr, 624 op/s 2024-04-02T04:47:22.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:22 smithi122 ceph-mon[27145]: Upgrade: Updating mgr.smithi122.cpwlyk 2024-04-02T04:47:22.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:22 smithi122 ceph-mon[27145]: Deploying daemon mgr.smithi122.cpwlyk on smithi122 2024-04-02T04:47:22.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:22 smithi122 ceph-mon[27145]: pgmap v19: 97 pgs: 97 active+clean; 341 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 6.7 MiB/s wr, 624 op/s 2024-04-02T04:47:23.318 INFO:teuthology.orchestra.run.smithi053.stdout: 5 136182 11.43 MB/sec execute 153 sec latency 54.987 ms 2024-04-02T04:47:24.096 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:24 smithi053 ceph-mon[24889]: pgmap v20: 97 pgs: 97 active+clean; 341 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.4 MiB/s wr, 439 op/s 2024-04-02T04:47:24.235 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:24 smithi122 ceph-mon[27145]: pgmap v20: 97 pgs: 97 active+clean; 341 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.4 MiB/s wr, 439 op/s 2024-04-02T04:47:24.318 INFO:teuthology.orchestra.run.smithi053.stdout: 5 136594 11.41 MB/sec execute 154 sec latency 70.279 ms 2024-04-02T04:47:25.318 INFO:teuthology.orchestra.run.smithi053.stdout: 5 137039 11.40 MB/sec execute 155 sec latency 34.961 ms 2024-04-02T04:47:26.318 INFO:teuthology.orchestra.run.smithi053.stdout: 5 137426 11.43 MB/sec execute 156 sec latency 34.046 ms 2024-04-02T04:47:26.375 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:26 smithi053 ceph-mon[24889]: pgmap v21: 97 pgs: 97 active+clean; 341 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.4 MiB/s wr, 439 op/s 2024-04-02T04:47:26.375 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:26 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:26.375 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:26 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:26.375 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:26 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:47:26.441 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:26 smithi122 ceph-mon[27145]: pgmap v21: 97 pgs: 97 active+clean; 341 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.4 MiB/s wr, 439 op/s 2024-04-02T04:47:26.441 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:26 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:26.441 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:26 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:26.442 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:26 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:47:27.318 INFO:teuthology.orchestra.run.smithi053.stdout: 5 137780 11.44 MB/sec execute 157 sec latency 49.731 ms 2024-04-02T04:47:27.641 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:27 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:27.642 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:27 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:27.642 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:27 smithi053 ceph-mon[24889]: pgmap v22: 97 pgs: 97 active+clean; 334 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.6 MiB/s wr, 679 op/s 2024-04-02T04:47:27.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:27 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:27.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:27 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:27.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:27 smithi122 ceph-mon[27145]: pgmap v22: 97 pgs: 97 active+clean; 334 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.6 MiB/s wr, 679 op/s 2024-04-02T04:47:28.318 INFO:teuthology.orchestra.run.smithi053.stdout: 5 138159 11.43 MB/sec execute 158 sec latency 54.860 ms 2024-04-02T04:47:29.318 INFO:teuthology.orchestra.run.smithi053.stdout: 5 138598 11.44 MB/sec execute 159 sec latency 32.649 ms 2024-04-02T04:47:30.318 INFO:teuthology.orchestra.run.smithi053.stdout: 5 139041 11.41 MB/sec execute 160 sec latency 24.888 ms 2024-04-02T04:47:30.438 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:30 smithi053 ceph-mon[24889]: pgmap v23: 97 pgs: 97 active+clean; 334 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.3 MiB/s wr, 423 op/s 2024-04-02T04:47:30.607 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:30 smithi122 ceph-mon[27145]: pgmap v23: 97 pgs: 97 active+clean; 334 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.3 MiB/s wr, 423 op/s 2024-04-02T04:47:31.318 INFO:teuthology.orchestra.run.smithi053.stdout: 5 139527 11.44 MB/sec execute 161 sec latency 22.939 ms 2024-04-02T04:47:31.951 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:31 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:31.951 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:31 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:31.951 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:31 smithi053 ceph-mon[24889]: pgmap v24: 97 pgs: 97 active+clean; 342 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 6.9 MiB/s wr, 671 op/s 2024-04-02T04:47:32.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:31 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:32.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:31 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:32.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:31 smithi122 ceph-mon[27145]: pgmap v24: 97 pgs: 97 active+clean; 342 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 6.9 MiB/s wr, 671 op/s 2024-04-02T04:47:32.318 INFO:teuthology.orchestra.run.smithi053.stdout: 5 140053 11.43 MB/sec execute 162 sec latency 29.874 ms 2024-04-02T04:47:33.319 INFO:teuthology.orchestra.run.smithi053.stdout: 5 140598 11.43 MB/sec execute 163 sec latency 31.283 ms 2024-04-02T04:47:33.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:33 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:33.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:33 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:33.463 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:33 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:33.463 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:33 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:34.319 INFO:teuthology.orchestra.run.smithi053.stdout: 5 141176 11.50 MB/sec execute 164 sec latency 42.039 ms 2024-04-02T04:47:34.423 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:34 smithi122 ceph-mon[27145]: pgmap v25: 97 pgs: 97 active+clean; 342 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.8 MiB/s wr, 487 op/s 2024-04-02T04:47:34.423 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:34 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:34.423 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:34 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:34.423 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:34 smithi122 ceph-mon[27145]: from='mgr.? 172.21.15.122:0/1458859164' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.cpwlyk/crt"}]: dispatch 2024-04-02T04:47:34.423 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:34 smithi122 ceph-mon[27145]: from='mgr.? 172.21.15.122:0/1458859164' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-02T04:47:34.423 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:34 smithi122 ceph-mon[27145]: from='mgr.? 172.21.15.122:0/1458859164' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.cpwlyk/key"}]: dispatch 2024-04-02T04:47:34.423 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:34 smithi122 ceph-mon[27145]: from='mgr.? 172.21.15.122:0/1458859164' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-02T04:47:34.423 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:34 smithi122 ceph-mon[27145]: Standby manager daemon smithi122.cpwlyk restarted 2024-04-02T04:47:34.423 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:34 smithi122 ceph-mon[27145]: Standby manager daemon smithi122.cpwlyk started 2024-04-02T04:47:34.476 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:34 smithi053 ceph-mon[24889]: pgmap v25: 97 pgs: 97 active+clean; 342 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.8 MiB/s wr, 487 op/s 2024-04-02T04:47:34.476 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:34 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:34.476 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:34 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:34.476 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:34 smithi053 ceph-mon[24889]: from='mgr.? 172.21.15.122:0/1458859164' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.cpwlyk/crt"}]: dispatch 2024-04-02T04:47:34.476 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:34 smithi053 ceph-mon[24889]: from='mgr.? 172.21.15.122:0/1458859164' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-02T04:47:34.476 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:34 smithi053 ceph-mon[24889]: from='mgr.? 172.21.15.122:0/1458859164' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.cpwlyk/key"}]: dispatch 2024-04-02T04:47:34.476 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:34 smithi053 ceph-mon[24889]: from='mgr.? 172.21.15.122:0/1458859164' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-02T04:47:34.476 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:34 smithi053 ceph-mon[24889]: Standby manager daemon smithi122.cpwlyk restarted 2024-04-02T04:47:34.476 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:34 smithi053 ceph-mon[24889]: Standby manager daemon smithi122.cpwlyk started 2024-04-02T04:47:35.319 INFO:teuthology.orchestra.run.smithi053.stdout: 5 141717 11.52 MB/sec execute 165 sec latency 59.230 ms 2024-04-02T04:47:36.248 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:36 smithi053 ceph-mon[24889]: mgrmap e33: smithi053.legdmv(active, since 45s), standbys: smithi122.cpwlyk 2024-04-02T04:47:36.248 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:36 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:47:36.248 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:36 smithi053 ceph-mon[24889]: pgmap v26: 97 pgs: 97 active+clean; 342 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.8 MiB/s wr, 487 op/s 2024-04-02T04:47:36.281 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:36 smithi122 ceph-mon[27145]: mgrmap e33: smithi053.legdmv(active, since 45s), standbys: smithi122.cpwlyk 2024-04-02T04:47:36.281 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:36 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:47:36.281 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:36 smithi122 ceph-mon[27145]: pgmap v26: 97 pgs: 97 active+clean; 342 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.8 MiB/s wr, 487 op/s 2024-04-02T04:47:36.319 INFO:teuthology.orchestra.run.smithi053.stdout: 5 142146 11.54 MB/sec execute 166 sec latency 41.076 ms 2024-04-02T04:47:37.319 INFO:teuthology.orchestra.run.smithi053.stdout: 5 142522 11.51 MB/sec execute 167 sec latency 54.025 ms 2024-04-02T04:47:37.521 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:37 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:37.521 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:37 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:37.521 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:37 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:47:37.521 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:37 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:47:37.521 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:37 smithi053 ceph-mon[24889]: pgmap v27: 97 pgs: 97 active+clean; 341 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.8 MiB/s wr, 867 op/s 2024-04-02T04:47:37.521 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:37 smithi053 ceph-mon[24889]: pgmap v28: 97 pgs: 97 active+clean; 341 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.9 MiB/s wr, 741 op/s 2024-04-02T04:47:37.521 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:37 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:37.521 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:37 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:37.521 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:37 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:47:37.521 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:37 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:47:37.522 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:37 smithi053 ceph-mon[24889]: Upgrade: Setting container_image for all mgr 2024-04-02T04:47:37.522 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:37 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:37.522 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:37 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi053.legdmv"}]: dispatch 2024-04-02T04:47:37.522 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:37 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi053.legdmv"}]': finished 2024-04-02T04:47:37.522 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:37 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi122.cpwlyk"}]: dispatch 2024-04-02T04:47:37.522 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:37 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi122.cpwlyk"}]': finished 2024-04-02T04:47:37.522 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:37 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-02T04:47:37.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:37 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:37.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:37 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:37.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:37 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:47:37.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:37 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:47:37.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:37 smithi122 ceph-mon[27145]: pgmap v27: 97 pgs: 97 active+clean; 341 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.8 MiB/s wr, 867 op/s 2024-04-02T04:47:37.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:37 smithi122 ceph-mon[27145]: pgmap v28: 97 pgs: 97 active+clean; 341 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.9 MiB/s wr, 741 op/s 2024-04-02T04:47:37.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:37 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:37.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:37 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:37.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:37 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:47:37.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:37 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:47:37.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:37 smithi122 ceph-mon[27145]: Upgrade: Setting container_image for all mgr 2024-04-02T04:47:37.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:37 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:37.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:37 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi053.legdmv"}]: dispatch 2024-04-02T04:47:37.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:37 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi053.legdmv"}]': finished 2024-04-02T04:47:37.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:37 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi122.cpwlyk"}]: dispatch 2024-04-02T04:47:37.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:37 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi122.cpwlyk"}]': finished 2024-04-02T04:47:37.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:37 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-02T04:47:38.319 INFO:teuthology.orchestra.run.smithi053.stdout: 5 143046 11.52 MB/sec execute 168 sec latency 44.290 ms 2024-04-02T04:47:39.289 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:39 smithi053 ceph-mon[24889]: pgmap v29: 97 pgs: 97 active+clean; 341 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.9 MiB/s wr, 741 op/s 2024-04-02T04:47:39.319 INFO:teuthology.orchestra.run.smithi053.stdout: 5 143541 11.52 MB/sec execute 169 sec latency 49.915 ms 2024-04-02T04:47:39.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:39 smithi122 ceph-mon[27145]: pgmap v29: 97 pgs: 97 active+clean; 341 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.9 MiB/s wr, 741 op/s 2024-04-02T04:47:40.319 INFO:teuthology.orchestra.run.smithi053.stdout: 5 144014 11.50 MB/sec execute 170 sec latency 37.972 ms 2024-04-02T04:47:40.545 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:40 smithi053 ceph-mon[24889]: Upgrade: Updating mon.smithi053 2024-04-02T04:47:40.546 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:40 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:40.546 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:40 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T04:47:40.546 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:40 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-02T04:47:40.546 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:40 smithi053 ceph-mon[24889]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:47:40.546 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:40 smithi053 ceph-mon[24889]: Deploying daemon mon.smithi053 on smithi053 2024-04-02T04:47:40.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:40 smithi122 ceph-mon[27145]: Upgrade: Updating mon.smithi053 2024-04-02T04:47:40.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:40 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' 2024-04-02T04:47:40.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:40 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T04:47:40.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:40 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-02T04:47:40.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:40 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:47:40.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:40 smithi122 ceph-mon[27145]: Deploying daemon mon.smithi053 on smithi053 2024-04-02T04:47:41.319 INFO:teuthology.orchestra.run.smithi053.stdout: 5 144489 11.54 MB/sec execute 171 sec latency 28.872 ms 2024-04-02T04:47:41.569 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:41 smithi053 ceph-mon[24889]: pgmap v30: 97 pgs: 97 active+clean; 341 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.4 MiB/s wr, 412 op/s 2024-04-02T04:47:41.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:41 smithi122 ceph-mon[27145]: pgmap v30: 97 pgs: 97 active+clean; 341 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.4 MiB/s wr, 412 op/s 2024-04-02T04:47:42.319 INFO:teuthology.orchestra.run.smithi053.stdout: 5 144988 11.57 MB/sec execute 172 sec latency 64.290 ms 2024-04-02T04:47:42.771 INFO:teuthology.orchestra.run.smithi053.stdout:true 2024-04-02T04:47:42.846 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:42 smithi053 systemd[1]: Stopping Ceph mon.smithi053 for 48e16742-f0aa-11ee-b647-cb9ed24678a4... 2024-04-02T04:47:43.319 INFO:teuthology.orchestra.run.smithi053.stdout: 5 145430 11.58 MB/sec execute 173 sec latency 42.321 ms 2024-04-02T04:47:43.351 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:43 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi053[24885]: 2024-04-02T04:47:43.146+0000 7fecc7e44700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi053 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T04:47:43.351 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:43 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi053[24885]: 2024-04-02T04:47:43.146+0000 7fecc7e44700 -1 mon.smithi053@0(leader) e2 *** Got Signal Terminated *** 2024-04-02T04:47:43.351 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:43 smithi053 podman[66933]: 2024-04-02 04:47:43.189743691 +0000 UTC m=+0.182288750 container died 665e4412f6da2eeedef9bdd05619b05b9dad0aaa37b331337a97a7b66b7684b0 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi053, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, name=centos-stream, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, maintainer=Guillaume Abrioux , io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, vcs-type=git, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., version=8, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD) 2024-04-02T04:47:43.925 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:43 smithi053 podman[66933]: 2024-04-02 04:47:43.878213866 +0000 UTC m=+0.870758919 container cleanup 665e4412f6da2eeedef9bdd05619b05b9dad0aaa37b331337a97a7b66b7684b0 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi053, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, com.redhat.component=centos-stream-container, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, ceph=True, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-02T04:47:43.925 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:43 smithi053 bash[66933]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi053 2024-04-02T04:47:43.943 INFO:teuthology.orchestra.run.smithi053.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-02T04:47:43.943 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager.smithi053 smithi053 *:9093,9094 running (2m) 9s ago 11m 25.7M - 0.23.0 ba2b418f427c bf50c1cd2b3d 2024-04-02T04:47:43.943 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi053 smithi053 running (11m) 9s ago 11m 6409k - 17.2.0 e1d6a67b021e a5c436bbd36d 2024-04-02T04:47:43.943 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi122 smithi122 running (10m) 13s ago 10m 6405k - 17.2.0 e1d6a67b021e b7074474cd46 2024-04-02T04:47:43.943 INFO:teuthology.orchestra.run.smithi053.stdout:grafana.smithi053 smithi053 *:3000 unknown 9s ago 11m - - 2024-04-02T04:47:43.943 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi053.nfrfot smithi053 running (5m) 9s ago 5m 36.5M - 17.2.0 e1d6a67b021e c94e9e643514 2024-04-02T04:47:43.943 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi122.mwoyoh smithi122 running (5m) 13s ago 5m 15.1M - 17.2.0 e1d6a67b021e 73376d9ee4b8 2024-04-02T04:47:43.943 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi053.legdmv smithi053 *:8443,9283,8765 running (77s) 9s ago 12m 494M - 19.0.0-1979-g6cec793d 78f830190839 c12d516a31bc 2024-04-02T04:47:43.943 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi122.cpwlyk smithi122 *:8443,9283,8765 running (18s) 13s ago 10m 95.9M - 19.0.0-1979-g6cec793d 78f830190839 9a545c5e7d81 2024-04-02T04:47:43.943 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi053 smithi053 running (12m) 9s ago 12m 55.6M 2048M 17.2.0 e1d6a67b021e 665e4412f6da 2024-04-02T04:47:43.943 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi122 smithi122 running (10m) 13s ago 10m 44.0M 2048M 17.2.0 e1d6a67b021e a2df07e0883d 2024-04-02T04:47:43.944 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.0.0.smithi053.mhkdcb smithi053 *:2049 running (5m) 9s ago 5m 265M - 4.0 e1d6a67b021e 07e5da7bd2c5 2024-04-02T04:47:43.944 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.1.0.smithi122.mauvsb smithi122 *:2049 running (5m) 13s ago 5m 81.1M - 4.0 e1d6a67b021e 4f866685005e 2024-04-02T04:47:43.944 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi053 smithi053 *:9100 running (113s) 9s ago 10m 20.6M - 1.3.1 1dbe0e931976 773b71971e67 2024-04-02T04:47:43.944 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi122 smithi122 *:9100 running (87s) 13s ago 10m 13.8M - 1.3.1 1dbe0e931976 475d439c7980 2024-04-02T04:47:43.944 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 smithi053 running (9m) 9s ago 9m 491M 1937M 17.2.0 e1d6a67b021e 906cb1fd8042 2024-04-02T04:47:43.944 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 smithi053 running (9m) 9s ago 9m 424M 1937M 17.2.0 e1d6a67b021e ac1ad6266e63 2024-04-02T04:47:43.944 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 smithi053 running (8m) 9s ago 8m 335M 1937M 17.2.0 e1d6a67b021e 86964c8a94ff 2024-04-02T04:47:43.944 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 smithi053 running (8m) 9s ago 8m 447M 1937M 17.2.0 e1d6a67b021e 7e8b9e98ae2b 2024-04-02T04:47:43.944 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 smithi122 running (8m) 13s ago 8m 484M 2704M 17.2.0 e1d6a67b021e 46076afcd7bc 2024-04-02T04:47:43.944 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 smithi122 running (7m) 13s ago 7m 522M 2704M 17.2.0 e1d6a67b021e e687a2f6da4d 2024-04-02T04:47:43.944 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 smithi122 running (7m) 13s ago 7m 435M 2704M 17.2.0 e1d6a67b021e 9e371df9a839 2024-04-02T04:47:43.944 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 smithi122 running (6m) 13s ago 6m 571M 2704M 17.2.0 e1d6a67b021e 9b3cc398254d 2024-04-02T04:47:43.944 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus.smithi053 smithi053 *:9095 running (25s) 9s ago 10m 50.1M - 2.33.4 514e6a882f6e 1dd55399da10 2024-04-02T04:47:44.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:44 smithi053 podman[66945]: 2024-04-02 04:47:44.167267157 +0000 UTC m=+0.979633860 container remove 665e4412f6da2eeedef9bdd05619b05b9dad0aaa37b331337a97a7b66b7684b0 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi053, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, architecture=x86_64, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, vcs-type=git, io.openshift.expose-services=, version=8, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux ) 2024-04-02T04:47:44.319 INFO:teuthology.orchestra.run.smithi053.stdout: 5 145807 11.57 MB/sec execute 174 sec latency 39.002 ms 2024-04-02T04:47:44.438 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:44 smithi053 systemd[1]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@mon.smithi053.service: Deactivated successfully. 2024-04-02T04:47:44.438 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:44 smithi053 systemd[1]: Stopped Ceph mon.smithi053 for 48e16742-f0aa-11ee-b647-cb9ed24678a4. 2024-04-02T04:47:44.438 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:44 smithi053 systemd[1]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@mon.smithi053.service: Consumed 16.868s CPU time. 2024-04-02T04:47:44.603 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:47:44.603 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": { 2024-04-02T04:47:44.603 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:47:44.603 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:47:44.603 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": { 2024-04-02T04:47:44.603 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:47:44.603 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:47:44.603 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": { 2024-04-02T04:47:44.604 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-02T04:47:44.604 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:47:44.604 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": { 2024-04-02T04:47:44.604 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:47:44.604 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:47:44.604 INFO:teuthology.orchestra.run.smithi053.stdout: "overall": { 2024-04-02T04:47:44.604 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-04-02T04:47:44.604 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:47:44.604 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-02T04:47:44.604 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:47:44.952 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:44 smithi053 systemd[1]: Starting Ceph mon.smithi053 for 48e16742-f0aa-11ee-b647-cb9ed24678a4... 2024-04-02T04:47:45.128 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:47:45.128 INFO:teuthology.orchestra.run.smithi053.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae", 2024-04-02T04:47:45.128 INFO:teuthology.orchestra.run.smithi053.stdout: "in_progress": true, 2024-04-02T04:47:45.128 INFO:teuthology.orchestra.run.smithi053.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-02T04:47:45.128 INFO:teuthology.orchestra.run.smithi053.stdout: "services_complete": [ 2024-04-02T04:47:45.128 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr" 2024-04-02T04:47:45.128 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-04-02T04:47:45.128 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": "2/23 daemons upgraded", 2024-04-02T04:47:45.128 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "Currently upgrading mon daemons", 2024-04-02T04:47:45.128 INFO:teuthology.orchestra.run.smithi053.stdout: "is_paused": false 2024-04-02T04:47:45.128 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:47:45.213 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 podman[67139]: 2024-04-02 04:47:44.931909882 +0000 UTC m=+0.027323096 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:47:45.213 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 podman[67139]: 2024-04-02 04:47:45.118822013 +0000 UTC m=+0.214235202 container create f53a44f19c00fe0a4b3a7509bf212ae21125b103873d1bf43002f1ef2a420368 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi053, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, RELEASE=squid-6cec793, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-02T04:47:45.319 INFO:teuthology.orchestra.run.smithi053.stdout: 5 146141 11.53 MB/sec execute 175 sec latency 62.151 ms 2024-04-02T04:47:45.718 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 podman[67139]: 2024-04-02 04:47:45.598955531 +0000 UTC m=+0.694368748 container init f53a44f19c00fe0a4b3a7509bf212ae21125b103873d1bf43002f1ef2a420368 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi053, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-6cec793, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-02T04:47:45.718 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 podman[67139]: 2024-04-02 04:47:45.611922231 +0000 UTC m=+0.707335426 container start f53a44f19c00fe0a4b3a7509bf212ae21125b103873d1bf43002f1ef2a420368 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi053, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-6cec793) 2024-04-02T04:47:45.718 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: set uid:gid to 167:167 (ceph:ceph) 2024-04-02T04:47:45.718 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev), process ceph-mon, pid 2 2024-04-02T04:47:45.718 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: pidfile_write: ignore empty --pid-file 2024-04-02T04:47:45.719 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: load: jerasure load: lrc 2024-04-02T04:47:45.719 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: RocksDB version: 7.9.2 2024-04-02T04:47:45.719 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Git sha 0 2024-04-02T04:47:45.719 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Compile date 2024-04-01 15:23:26 2024-04-02T04:47:45.719 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: DB SUMMARY 2024-04-02T04:47:45.719 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: DB Session ID: YGD306TAAAA8KCZ6MTKH 2024-04-02T04:47:45.719 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: CURRENT file: CURRENT 2024-04-02T04:47:45.719 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: IDENTITY file: IDENTITY 2024-04-02T04:47:45.719 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: MANIFEST file: MANIFEST-000015 size: 978 Bytes 2024-04-02T04:47:45.719 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi053/store.db dir, Total Num: 1, files: 000030.sst 2024-04-02T04:47:45.719 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi053/store.db: 000028.log size: 2732785 ; 2024-04-02T04:47:45.719 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.error_if_exists: 0 2024-04-02T04:47:45.719 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.create_if_missing: 0 2024-04-02T04:47:45.719 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.paranoid_checks: 1 2024-04-02T04:47:45.720 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-02T04:47:45.720 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-02T04:47:45.720 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-02T04:47:45.720 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.env: 0x55d1e4910c60 2024-04-02T04:47:45.720 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.fs: PosixFileSystem 2024-04-02T04:47:45.720 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.info_log: 0x55d1e66619a0 2024-04-02T04:47:45.720 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.max_file_opening_threads: 16 2024-04-02T04:47:45.721 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.statistics: (nil) 2024-04-02T04:47:45.721 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.use_fsync: 0 2024-04-02T04:47:45.721 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.max_log_file_size: 0 2024-04-02T04:47:45.721 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-02T04:47:45.722 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-02T04:47:45.722 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.keep_log_file_num: 1000 2024-04-02T04:47:45.722 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.recycle_log_file_num: 0 2024-04-02T04:47:45.722 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.allow_fallocate: 1 2024-04-02T04:47:45.722 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.allow_mmap_reads: 0 2024-04-02T04:47:45.722 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.allow_mmap_writes: 0 2024-04-02T04:47:45.722 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.use_direct_reads: 0 2024-04-02T04:47:45.722 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-02T04:47:45.722 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.create_missing_column_families: 0 2024-04-02T04:47:45.722 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.db_log_dir: 2024-04-02T04:47:45.722 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.wal_dir: 2024-04-02T04:47:45.722 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-02T04:47:45.722 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-02T04:47:45.722 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-02T04:47:45.722 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-02T04:47:45.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-02T04:47:45.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-02T04:47:45.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.advise_random_on_open: 1 2024-04-02T04:47:45.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.db_write_buffer_size: 0 2024-04-02T04:47:45.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.write_buffer_manager: 0x55d1e66341e0 2024-04-02T04:47:45.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-02T04:47:45.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-02T04:47:45.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-02T04:47:45.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.rate_limiter: (nil) 2024-04-02T04:47:45.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-02T04:47:45.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.wal_recovery_mode: 2 2024-04-02T04:47:45.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.enable_thread_tracking: 0 2024-04-02T04:47:45.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.enable_pipelined_write: 0 2024-04-02T04:47:45.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.unordered_write: 0 2024-04-02T04:47:45.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-02T04:47:45.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-02T04:47:45.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-02T04:47:45.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-02T04:47:45.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.row_cache: None 2024-04-02T04:47:45.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.wal_filter: None 2024-04-02T04:47:45.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-02T04:47:45.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.allow_ingest_behind: 0 2024-04-02T04:47:45.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.two_write_queues: 0 2024-04-02T04:47:45.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.manual_wal_flush: 0 2024-04-02T04:47:45.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.wal_compression: 0 2024-04-02T04:47:45.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.atomic_flush: 0 2024-04-02T04:47:45.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-02T04:47:45.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-02T04:47:45.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-02T04:47:45.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.log_readahead_size: 0 2024-04-02T04:47:45.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-02T04:47:45.725 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.best_efforts_recovery: 0 2024-04-02T04:47:45.725 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-02T04:47:45.725 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-02T04:47:45.725 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.allow_data_in_errors: 0 2024-04-02T04:47:45.725 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.db_host_id: __hostname__ 2024-04-02T04:47:45.725 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-02T04:47:45.725 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.max_background_jobs: 2 2024-04-02T04:47:45.725 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.max_background_compactions: -1 2024-04-02T04:47:45.725 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.max_subcompactions: 1 2024-04-02T04:47:45.725 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-02T04:47:45.725 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-02T04:47:45.725 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-02T04:47:45.725 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.max_total_wal_size: 0 2024-04-02T04:47:45.725 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-02T04:47:45.725 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-02T04:47:45.726 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-02T04:47:45.726 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-02T04:47:45.726 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.max_open_files: -1 2024-04-02T04:47:45.726 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.bytes_per_sync: 0 2024-04-02T04:47:45.726 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-02T04:47:45.726 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-02T04:47:45.726 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.compaction_readahead_size: 0 2024-04-02T04:47:45.726 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.max_background_flushes: -1 2024-04-02T04:47:45.726 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Compression algorithms supported: 2024-04-02T04:47:45.726 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-02T04:47:45.726 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: kZSTD supported: 0 2024-04-02T04:47:45.726 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: kXpressCompression supported: 0 2024-04-02T04:47:45.726 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: kLZ4HCCompression supported: 1 2024-04-02T04:47:45.726 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: kZlibCompression supported: 1 2024-04-02T04:47:45.727 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: kSnappyCompression supported: 1 2024-04-02T04:47:45.727 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: kLZ4Compression supported: 1 2024-04-02T04:47:45.727 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: kBZip2Compression supported: 0 2024-04-02T04:47:45.727 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-02T04:47:45.727 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-02T04:47:45.727 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi053/store.db/MANIFEST-000015 2024-04-02T04:47:45.727 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-02T04:47:45.727 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-02T04:47:45.727 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.merge_operator: 2024-04-02T04:47:45.727 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.compaction_filter: None 2024-04-02T04:47:45.727 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.compaction_filter_factory: None 2024-04-02T04:47:45.727 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.sst_partitioner_factory: None 2024-04-02T04:47:45.727 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-02T04:47:45.727 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-02T04:47:45.727 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d1e6661aa0) 2024-04-02T04:47:45.727 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: cache_index_and_filter_blocks: 1 2024-04-02T04:47:45.728 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-02T04:47:45.728 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-02T04:47:45.728 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: pin_top_level_index_and_filter: 1 2024-04-02T04:47:45.728 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: index_type: 0 2024-04-02T04:47:45.728 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: data_block_index_type: 0 2024-04-02T04:47:45.728 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: index_shortening: 1 2024-04-02T04:47:45.728 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-02T04:47:45.728 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: checksum: 4 2024-04-02T04:47:45.728 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: no_block_cache: 0 2024-04-02T04:47:45.728 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: block_cache: 0x55d1e6665090 2024-04-02T04:47:45.728 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: block_cache_name: BinnedLRUCache 2024-04-02T04:47:45.728 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: block_cache_options: 2024-04-02T04:47:45.728 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: capacity : 536870912 2024-04-02T04:47:45.728 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: num_shard_bits : 4 2024-04-02T04:47:45.728 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: strict_capacity_limit : 0 2024-04-02T04:47:45.728 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: high_pri_pool_ratio: 0.000 2024-04-02T04:47:45.729 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: block_cache_compressed: (nil) 2024-04-02T04:47:45.729 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: persistent_cache: (nil) 2024-04-02T04:47:45.729 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: block_size: 4096 2024-04-02T04:47:45.729 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: block_size_deviation: 10 2024-04-02T04:47:45.729 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: block_restart_interval: 16 2024-04-02T04:47:45.729 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: index_block_restart_interval: 1 2024-04-02T04:47:45.729 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: metadata_block_size: 4096 2024-04-02T04:47:45.729 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: partition_filters: 0 2024-04-02T04:47:45.729 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: use_delta_encoding: 1 2024-04-02T04:47:45.729 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: filter_policy: bloomfilter 2024-04-02T04:47:45.729 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: whole_key_filtering: 1 2024-04-02T04:47:45.729 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: verify_compression: 0 2024-04-02T04:47:45.729 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: read_amp_bytes_per_bit: 0 2024-04-02T04:47:45.729 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: format_version: 5 2024-04-02T04:47:45.730 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: enable_index_compression: 1 2024-04-02T04:47:45.730 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: block_align: 0 2024-04-02T04:47:45.730 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: max_auto_readahead_size: 262144 2024-04-02T04:47:45.730 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: prepopulate_block_cache: 0 2024-04-02T04:47:45.730 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: initial_auto_readahead_size: 8192 2024-04-02T04:47:45.730 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: num_file_reads_for_auto_readahead: 2 2024-04-02T04:47:45.730 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.write_buffer_size: 33554432 2024-04-02T04:47:45.730 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.max_write_buffer_number: 2 2024-04-02T04:47:45.730 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.compression: NoCompression 2024-04-02T04:47:45.730 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.bottommost_compression: Disabled 2024-04-02T04:47:45.730 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.prefix_extractor: nullptr 2024-04-02T04:47:45.730 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-02T04:47:45.730 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.num_levels: 7 2024-04-02T04:47:45.730 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-02T04:47:45.730 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-02T04:47:45.731 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-02T04:47:45.731 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-02T04:47:45.731 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-02T04:47:45.731 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-02T04:47:45.731 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-02T04:47:45.731 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-02T04:47:45.731 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-02T04:47:45.731 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-02T04:47:45.731 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-02T04:47:45.731 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-02T04:47:45.731 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-02T04:47:45.731 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.compression_opts.level: 32767 2024-04-02T04:47:45.731 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.compression_opts.strategy: 0 2024-04-02T04:47:45.731 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-02T04:47:45.731 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-02T04:47:45.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-02T04:47:45.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-02T04:47:45.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.compression_opts.enabled: false 2024-04-02T04:47:45.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-02T04:47:45.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-02T04:47:45.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-02T04:47:45.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-02T04:47:45.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.target_file_size_base: 67108864 2024-04-02T04:47:45.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-02T04:47:45.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-02T04:47:45.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-02T04:47:45.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-02T04:47:45.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-02T04:47:45.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-02T04:47:45.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-02T04:47:45.733 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-02T04:47:45.733 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-02T04:47:45.733 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-02T04:47:45.733 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-02T04:47:45.733 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-02T04:47:45.733 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-02T04:47:45.733 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-02T04:47:45.733 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.arena_block_size: 1048576 2024-04-02T04:47:45.733 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-02T04:47:45.733 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-02T04:47:45.733 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.disable_auto_compactions: 0 2024-04-02T04:47:45.733 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-02T04:47:45.733 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-02T04:47:45.733 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-02T04:47:45.733 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-02T04:47:45.734 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-02T04:47:45.734 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-02T04:47:45.734 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-02T04:47:45.734 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-02T04:47:45.734 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-02T04:47:45.734 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-02T04:47:45.734 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.table_properties_collectors: 2024-04-02T04:47:45.734 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.inplace_update_support: 0 2024-04-02T04:47:45.734 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-02T04:47:45.734 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-02T04:47:45.734 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-02T04:47:45.734 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-02T04:47:45.734 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.bloom_locality: 0 2024-04-02T04:47:45.734 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.max_successive_merges: 0 2024-04-02T04:47:45.734 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-02T04:47:45.735 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.paranoid_file_checks: 0 2024-04-02T04:47:45.735 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.force_consistency_checks: 1 2024-04-02T04:47:45.735 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.report_bg_io_stats: 0 2024-04-02T04:47:45.735 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.ttl: 2592000 2024-04-02T04:47:45.735 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-02T04:47:45.735 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-02T04:47:45.735 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-02T04:47:45.735 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.enable_blob_files: false 2024-04-02T04:47:45.735 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.min_blob_size: 0 2024-04-02T04:47:45.735 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.blob_file_size: 268435456 2024-04-02T04:47:45.735 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-02T04:47:45.735 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-02T04:47:45.735 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-02T04:47:45.735 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-02T04:47:45.735 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-02T04:47:45.736 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.blob_file_starting_level: 0 2024-04-02T04:47:45.736 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-02T04:47:45.736 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-04-02T04:47:45.736 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-02T04:47:45.736 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi053/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 11799, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-02T04:47:45.736 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-04-02T04:47:45.736 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 9c22fed5-2c67-4d91-9b93-3b43a6c7a70b 2024-04-02T04:47:45.736 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712033265647241, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-04-02T04:47:45.736 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-04-02T04:47:45.736 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712033265658595, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 2498543, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11800, "largest_seqno": 13024, "table_properties": {"data_size": 2492539, "index_size": 3641, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1349, "raw_key_size": 11797, "raw_average_key_size": 22, "raw_value_size": 2481266, "raw_average_value_size": 4817, "num_data_blocks": 166, "num_entries": 515, "num_filter_entries": 515, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712033265, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9c22fed5-2c67-4d91-9b93-3b43a6c7a70b", "db_session_id": "YGD306TAAAA8KCZ6MTKH", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-04-02T04:47:45.736 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712033265658787, "job": 1, "event": "recovery_finished"} 2024-04-02T04:47:45.736 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-04-02T04:47:45.736 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-02T04:47:45.736 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi053/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-02T04:47:45.736 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55d1e6756000 2024-04-02T04:47:45.737 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: DB pointer 0x55d1e6740000 2024-04-02T04:47:45.737 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-02T04:47:45.737 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-02T04:47:45.737 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: ** DB Stats ** 2024-04-02T04:47:45.737 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-02T04:47:45.737 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-02T04:47:45.737 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-02T04:47:45.737 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-02T04:47:45.737 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-02T04:47:45.737 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-02T04:47:45.737 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-02T04:47:45.737 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: 2024-04-02T04:47:45.737 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: ** Compaction Stats [default] ** 2024-04-02T04:47:45.737 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-02T04:47:45.738 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-02T04:47:45.738 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: L0 1/0 2.38 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 434.2 0.01 0.00 1 0.005 0 0 0.0 0.0 2024-04-02T04:47:45.738 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: L6 1/0 6.45 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-02T04:47:45.738 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Sum 2/0 8.83 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 434.2 0.01 0.00 1 0.005 0 0 0.0 0.0 2024-04-02T04:47:45.738 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 434.2 0.01 0.00 1 0.005 0 0 0.0 0.0 2024-04-02T04:47:45.738 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: 2024-04-02T04:47:45.738 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: ** Compaction Stats [default] ** 2024-04-02T04:47:45.738 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-02T04:47:45.738 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-02T04:47:45.738 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 434.2 0.01 0.00 1 0.005 0 0 0.0 0.0 2024-04-02T04:47:45.738 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: 2024-04-02T04:47:45.738 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-02T04:47:45.738 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: 2024-04-02T04:47:45.738 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-02T04:47:45.739 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-04-02T04:47:45.739 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-02T04:47:45.739 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-02T04:47:45.739 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-02T04:47:45.739 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-02T04:47:45.739 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Cumulative compaction: 0.00 GB write, 163.84 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-02T04:47:45.739 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Interval compaction: 0.00 GB write, 163.84 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-02T04:47:45.739 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-02T04:47:45.739 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Block cache BinnedLRUCache@0x55d1e6665090#2 capacity: 512.00 MB usage: 5.48 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-04-02T04:47:45.739 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.41 KB,0.000268221%) IndexBlock(1,4.08 KB,0.000777841%) Misc(1,0.00 KB,0%) 2024-04-02T04:47:45.739 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: 2024-04-02T04:47:45.739 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-02T04:47:45.739 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: starting mon.smithi053 rank 0 at public addrs [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] at bind addrs [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi053 fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 2024-04-02T04:47:45.739 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: mon.smithi053@-1(???) e2 preinit fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 2024-04-02T04:47:45.739 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: mon.smithi053@-1(???).mds e7 new map 2024-04-02T04:47:45.740 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: mon.smithi053@-1(???).mds e7 print_map 2024-04-02T04:47:45.740 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: e7 2024-04-02T04:47:45.740 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-02T04:47:45.740 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-02T04:47:45.740 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: legacy client fscid: 1 2024-04-02T04:47:45.740 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: 2024-04-02T04:47:45.740 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Filesystem 'foofs' (1) 2024-04-02T04:47:45.740 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: fs_name foofs 2024-04-02T04:47:45.740 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: epoch 5 2024-04-02T04:47:45.740 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-02T04:47:45.740 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: created 2024-04-02T04:42:12.430527+0000 2024-04-02T04:47:45.740 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: modified 2024-04-02T04:42:18.452666+0000 2024-04-02T04:47:45.740 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: tableserver 0 2024-04-02T04:47:45.740 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: root 0 2024-04-02T04:47:45.740 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: session_timeout 60 2024-04-02T04:47:45.740 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: session_autoclose 300 2024-04-02T04:47:45.741 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: max_file_size 1099511627776 2024-04-02T04:47:45.741 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: max_xattr_size 65536 2024-04-02T04:47:45.741 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: required_client_features {} 2024-04-02T04:47:45.741 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: last_failure 0 2024-04-02T04:47:45.741 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: last_failure_osd_epoch 0 2024-04-02T04:47:45.741 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-02T04:47:45.741 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: max_mds 1 2024-04-02T04:47:45.741 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: in 0 2024-04-02T04:47:45.741 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: up {0=14478} 2024-04-02T04:47:45.741 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: failed 2024-04-02T04:47:45.741 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: damaged 2024-04-02T04:47:45.741 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: stopped 2024-04-02T04:47:45.741 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: data_pools [3] 2024-04-02T04:47:45.741 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: metadata_pool 2 2024-04-02T04:47:45.741 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: inline_data disabled 2024-04-02T04:47:45.742 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: balancer 2024-04-02T04:47:45.742 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: bal_rank_mask -1 2024-04-02T04:47:45.742 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: standby_count_wanted 1 2024-04-02T04:47:45.742 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: qdb_cluster leader: 0 members: 2024-04-02T04:47:45.742 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: [mds.foofs.smithi053.nfrfot{0:14478} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.53:6834/2480061460,v1:172.21.15.53:6835/2480061460] compat {c=[1],r=[1],i=[7ff]}] 2024-04-02T04:47:45.742 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: 2024-04-02T04:47:45.742 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: 2024-04-02T04:47:45.742 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: Standby daemons: 2024-04-02T04:47:45.742 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: 2024-04-02T04:47:45.742 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout: [mds.foofs.smithi122.mwoyoh{-1:24301} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6832/233414887,v1:172.21.15.122:6833/233414887] compat {c=[1],r=[1],i=[7ff]}] 2024-04-02T04:47:45.742 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: mon.smithi053@-1(???).osd e71 crush map has features 3314933000854323200, adjusting msgr requires 2024-04-02T04:47:45.742 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: mon.smithi053@-1(???).osd e71 crush map has features 432629239337189376, adjusting msgr requires 2024-04-02T04:47:45.742 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: mon.smithi053@-1(???).osd e71 crush map has features 432629239337189376, adjusting msgr requires 2024-04-02T04:47:45.742 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: mon.smithi053@-1(???).osd e71 crush map has features 432629239337189376, adjusting msgr requires 2024-04-02T04:47:45.743 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 ceph-mon[67180]: mon.smithi053@-1(???).paxosservice(auth 1..29) refresh upgraded, format 0 -> 3 2024-04-02T04:47:45.836 INFO:teuthology.orchestra.run.smithi053.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-02T04:47:45.836 INFO:teuthology.orchestra.run.smithi053.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-02T04:47:45.836 INFO:teuthology.orchestra.run.smithi053.stdout: daemon grafana.smithi053 on smithi053 is in unknown state 2024-04-02T04:47:45.982 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 bash[67139]: f53a44f19c00fe0a4b3a7509bf212ae21125b103873d1bf43002f1ef2a420368 2024-04-02T04:47:45.983 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:45 smithi053 systemd[1]: Started Ceph mon.smithi053 for 48e16742-f0aa-11ee-b647-cb9ed24678a4. 2024-04-02T04:47:46.319 INFO:teuthology.orchestra.run.smithi053.stdout: 5 146450 11.52 MB/sec execute 176 sec latency 48.637 ms 2024-04-02T04:47:46.506 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:46 smithi053 ceph-mon[67180]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi053"}]: dispatch 2024-04-02T04:47:46.506 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:46 smithi053 ceph-mon[67180]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-02T04:47:46.506 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:46 smithi053 ceph-mon[67180]: from='client.14842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:47:46.506 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:46 smithi053 ceph-mon[67180]: from='client.24621 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:47:46.506 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:46 smithi053 ceph-mon[67180]: pgmap v32: 97 pgs: 97 active+clean; 335 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 7.9 MiB/s wr, 796 op/s 2024-04-02T04:47:46.507 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:46 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/4128135745' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:47:46.507 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:46 smithi053 ceph-mon[67180]: from='client.24629 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:47:46.507 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:46 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/4215974594' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:47:46.507 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:46 smithi053 ceph-mon[67180]: mon.smithi053 calling monitor election 2024-04-02T04:47:46.507 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:46 smithi053 ceph-mon[67180]: mon.smithi053 is new leader, mons smithi053,smithi122 in quorum (ranks 0,1) 2024-04-02T04:47:46.507 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:46 smithi053 ceph-mon[67180]: monmap epoch 2 2024-04-02T04:47:46.507 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:46 smithi053 ceph-mon[67180]: fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 2024-04-02T04:47:46.507 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:46 smithi053 ceph-mon[67180]: last_changed 2024-04-02T04:37:12.545751+0000 2024-04-02T04:47:46.507 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:46 smithi053 ceph-mon[67180]: created 2024-04-02T04:34:55.992242+0000 2024-04-02T04:47:46.507 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:46 smithi053 ceph-mon[67180]: min_mon_release 17 (quincy) 2024-04-02T04:47:46.507 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:46 smithi053 ceph-mon[67180]: election_strategy: 1 2024-04-02T04:47:46.507 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:46 smithi053 ceph-mon[67180]: 0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.smithi053 2024-04-02T04:47:46.507 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:46 smithi053 ceph-mon[67180]: 1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.smithi122 2024-04-02T04:47:46.507 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:46 smithi053 ceph-mon[67180]: fsmap foofs:1 {0=foofs.smithi053.nfrfot=up:active} 1 up:standby 2024-04-02T04:47:46.508 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:46 smithi053 ceph-mon[67180]: osdmap e71: 8 total, 8 up, 8 in 2024-04-02T04:47:46.508 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:46 smithi053 ceph-mon[67180]: mgrmap e33: smithi053.legdmv(active, since 57s), standbys: smithi122.cpwlyk 2024-04-02T04:47:46.508 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:46 smithi053 ceph-mon[67180]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-02T04:47:46.508 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:46 smithi053 ceph-mon[67180]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-02T04:47:46.508 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:46 smithi053 ceph-mon[67180]: daemon grafana.smithi053 on smithi053 is in unknown state 2024-04-02T04:47:46.508 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:46 smithi053 ceph-mon[67180]: from='mgr.24547 ' entity='' 2024-04-02T04:47:46.508 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:46 smithi053 ceph-mon[67180]: mgrmap e34: smithi053.legdmv(active, since 57s), standbys: smithi122.cpwlyk 2024-04-02T04:47:46.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:46 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi053"}]: dispatch 2024-04-02T04:47:46.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:46 smithi122 ceph-mon[27145]: from='mgr.24547 172.21.15.53:0/4195605105' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-02T04:47:46.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:46 smithi122 ceph-mon[27145]: from='client.14842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:47:46.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:46 smithi122 ceph-mon[27145]: from='client.24621 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:47:46.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:46 smithi122 ceph-mon[27145]: pgmap v32: 97 pgs: 97 active+clean; 335 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 7.9 MiB/s wr, 796 op/s 2024-04-02T04:47:46.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:46 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/4128135745' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:47:46.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:46 smithi122 ceph-mon[27145]: from='client.24629 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:47:46.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:46 smithi122 ceph-mon[27145]: from='client.? 172.21.15.53:0/4215974594' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:47:46.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:46 smithi122 ceph-mon[27145]: mon.smithi053 calling monitor election 2024-04-02T04:47:46.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:46 smithi122 ceph-mon[27145]: mon.smithi053 is new leader, mons smithi053,smithi122 in quorum (ranks 0,1) 2024-04-02T04:47:46.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:46 smithi122 ceph-mon[27145]: monmap epoch 2 2024-04-02T04:47:46.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:46 smithi122 ceph-mon[27145]: fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 2024-04-02T04:47:46.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:46 smithi122 ceph-mon[27145]: last_changed 2024-04-02T04:37:12.545751+0000 2024-04-02T04:47:46.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:46 smithi122 ceph-mon[27145]: created 2024-04-02T04:34:55.992242+0000 2024-04-02T04:47:46.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:46 smithi122 ceph-mon[27145]: min_mon_release 17 (quincy) 2024-04-02T04:47:46.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:46 smithi122 ceph-mon[27145]: election_strategy: 1 2024-04-02T04:47:46.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:46 smithi122 ceph-mon[27145]: 0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.smithi053 2024-04-02T04:47:46.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:46 smithi122 ceph-mon[27145]: 1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.smithi122 2024-04-02T04:47:46.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:46 smithi122 ceph-mon[27145]: fsmap foofs:1 {0=foofs.smithi053.nfrfot=up:active} 1 up:standby 2024-04-02T04:47:46.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:46 smithi122 ceph-mon[27145]: osdmap e71: 8 total, 8 up, 8 in 2024-04-02T04:47:46.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:46 smithi122 ceph-mon[27145]: mgrmap e33: smithi053.legdmv(active, since 57s), standbys: smithi122.cpwlyk 2024-04-02T04:47:46.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:46 smithi122 ceph-mon[27145]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-02T04:47:46.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:46 smithi122 ceph-mon[27145]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-02T04:47:46.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:46 smithi122 ceph-mon[27145]: daemon grafana.smithi053 on smithi053 is in unknown state 2024-04-02T04:47:46.897 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:46 smithi122 ceph-mon[27145]: from='mgr.24547 ' entity='' 2024-04-02T04:47:46.897 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:46 smithi122 ceph-mon[27145]: mgrmap e34: smithi053.legdmv(active, since 57s), standbys: smithi122.cpwlyk 2024-04-02T04:47:47.319 INFO:teuthology.orchestra.run.smithi053.stdout: 5 146732 11.49 MB/sec execute 177 sec latency 88.699 ms 2024-04-02T04:47:48.319 INFO:teuthology.orchestra.run.smithi053.stdout: 5 147162 11.48 MB/sec execute 178 sec latency 39.240 ms 2024-04-02T04:47:48.490 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:48 smithi122 ceph-mon[27145]: mgrmap e35: smithi053.legdmv(active, since 58s), standbys: smithi122.cpwlyk 2024-04-02T04:47:48.567 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:48 smithi053 ceph-mon[67180]: mgrmap e35: smithi053.legdmv(active, since 58s), standbys: smithi122.cpwlyk 2024-04-02T04:47:49.319 INFO:teuthology.orchestra.run.smithi053.stdout: 5 147675 11.47 MB/sec execute 179 sec latency 31.312 ms 2024-04-02T04:47:50.319 INFO:teuthology.orchestra.run.smithi053.stdout: 5 148151 11.50 MB/sec execute 180 sec latency 33.472 ms 2024-04-02T04:47:51.319 INFO:teuthology.orchestra.run.smithi053.stdout: 5 148691 11.53 MB/sec execute 181 sec latency 29.429 ms 2024-04-02T04:47:52.319 INFO:teuthology.orchestra.run.smithi053.stdout: 5 149051 11.54 MB/sec execute 182 sec latency 32.165 ms 2024-04-02T04:47:53.319 INFO:teuthology.orchestra.run.smithi053.stdout: 5 149344 11.52 MB/sec execute 183 sec latency 61.351 ms 2024-04-02T04:47:54.319 INFO:teuthology.orchestra.run.smithi053.stdout: 5 149633 11.48 MB/sec execute 184 sec latency 50.670 ms 2024-04-02T04:47:54.656 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:54 smithi053 ceph-mon[67180]: Standby manager daemon smithi122.cpwlyk restarted 2024-04-02T04:47:54.657 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:54 smithi053 ceph-mon[67180]: Standby manager daemon smithi122.cpwlyk started 2024-04-02T04:47:54.657 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:54 smithi053 ceph-mon[67180]: from='mgr.? 172.21.15.122:0/641808938' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.cpwlyk/crt"}]: dispatch 2024-04-02T04:47:54.657 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:54 smithi053 ceph-mon[67180]: from='mgr.? 172.21.15.122:0/641808938' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-02T04:47:54.657 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:54 smithi053 ceph-mon[67180]: from='mgr.? 172.21.15.122:0/641808938' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.cpwlyk/key"}]: dispatch 2024-04-02T04:47:54.657 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:54 smithi053 ceph-mon[67180]: from='mgr.? 172.21.15.122:0/641808938' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-02T04:47:54.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:54 smithi122 ceph-mon[27145]: Standby manager daemon smithi122.cpwlyk restarted 2024-04-02T04:47:54.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:54 smithi122 ceph-mon[27145]: Standby manager daemon smithi122.cpwlyk started 2024-04-02T04:47:54.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:54 smithi122 ceph-mon[27145]: from='mgr.? 172.21.15.122:0/641808938' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.cpwlyk/crt"}]: dispatch 2024-04-02T04:47:54.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:54 smithi122 ceph-mon[27145]: from='mgr.? 172.21.15.122:0/641808938' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-02T04:47:54.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:54 smithi122 ceph-mon[27145]: from='mgr.? 172.21.15.122:0/641808938' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.cpwlyk/key"}]: dispatch 2024-04-02T04:47:54.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:54 smithi122 ceph-mon[27145]: from='mgr.? 172.21.15.122:0/641808938' entity='mgr.smithi122.cpwlyk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-02T04:47:55.319 INFO:teuthology.orchestra.run.smithi053.stdout: 5 149938 11.47 MB/sec execute 185 sec latency 42.876 ms 2024-04-02T04:47:55.667 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:55 smithi053 ceph-mon[67180]: mgrmap e36: smithi053.legdmv(active, since 65s), standbys: smithi122.cpwlyk 2024-04-02T04:47:55.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:55 smithi122 ceph-mon[27145]: mgrmap e36: smithi053.legdmv(active, since 65s), standbys: smithi122.cpwlyk 2024-04-02T04:47:56.320 INFO:teuthology.orchestra.run.smithi053.stdout: 5 150218 11.44 MB/sec execute 186 sec latency 61.427 ms 2024-04-02T04:47:56.695 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:56 smithi053 ceph-mon[67180]: Active manager daemon smithi053.legdmv restarted 2024-04-02T04:47:56.695 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:56 smithi053 ceph-mon[67180]: Activating manager daemon smithi053.legdmv 2024-04-02T04:47:56.695 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:56 smithi053 ceph-mon[67180]: osdmap e72: 8 total, 8 up, 8 in 2024-04-02T04:47:56.695 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:56 smithi053 ceph-mon[67180]: mgrmap e37: smithi053.legdmv(active, starting, since 0.00684761s), standbys: smithi122.cpwlyk 2024-04-02T04:47:56.695 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:56 smithi053 ceph-mon[67180]: failed to encode map e72 with expected crc 2024-04-02T04:47:56.695 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:56 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi053"}]: dispatch 2024-04-02T04:47:56.695 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:56 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-02T04:47:56.696 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:56 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi053.nfrfot"}]: dispatch 2024-04-02T04:47:56.696 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:56 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi122.mwoyoh"}]: dispatch 2024-04-02T04:47:56.696 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:56 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mgr metadata", "who": "smithi053.legdmv", "id": "smithi053.legdmv"}]: dispatch 2024-04-02T04:47:56.696 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:56 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mgr metadata", "who": "smithi122.cpwlyk", "id": "smithi122.cpwlyk"}]: dispatch 2024-04-02T04:47:56.696 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:56 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:47:56.696 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:56 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:47:56.696 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:56 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:47:56.696 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:56 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:47:56.696 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:56 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:47:56.696 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:56 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:47:56.696 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:56 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:47:56.696 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:56 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:47:56.696 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:56 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-02T04:47:56.696 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:56 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-02T04:47:56.697 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:56 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-02T04:47:56.697 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:56 smithi053 ceph-mon[67180]: Manager daemon smithi053.legdmv is now available 2024-04-02T04:47:56.828 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:56 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi122[27128]: 2024-04-02T04:47:56.494+0000 7f0721916700 -1 mon.smithi122@1(peon).osd e72 update_from_paxos full map CRC mismatch, resetting to canonical 2024-04-02T04:47:56.829 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:56 smithi122 ceph-mon[27145]: Active manager daemon smithi053.legdmv restarted 2024-04-02T04:47:56.829 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:56 smithi122 ceph-mon[27145]: Activating manager daemon smithi053.legdmv 2024-04-02T04:47:56.829 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:56 smithi122 ceph-mon[27145]: osdmap e72: 8 total, 8 up, 8 in 2024-04-02T04:47:56.829 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:56 smithi122 ceph-mon[27145]: mgrmap e37: smithi053.legdmv(active, starting, since 0.00684761s), standbys: smithi122.cpwlyk 2024-04-02T04:47:56.829 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:56 smithi122 ceph-mon[27145]: failed to encode map e72 with expected crc 2024-04-02T04:47:56.829 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:56 smithi122 ceph-mon[27145]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi053"}]: dispatch 2024-04-02T04:47:56.829 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:56 smithi122 ceph-mon[27145]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-02T04:47:56.829 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:56 smithi122 ceph-mon[27145]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi053.nfrfot"}]: dispatch 2024-04-02T04:47:56.829 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:56 smithi122 ceph-mon[27145]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi122.mwoyoh"}]: dispatch 2024-04-02T04:47:56.829 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:56 smithi122 ceph-mon[27145]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mgr metadata", "who": "smithi053.legdmv", "id": "smithi053.legdmv"}]: dispatch 2024-04-02T04:47:56.829 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:56 smithi122 ceph-mon[27145]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mgr metadata", "who": "smithi122.cpwlyk", "id": "smithi122.cpwlyk"}]: dispatch 2024-04-02T04:47:56.829 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:56 smithi122 ceph-mon[27145]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:47:56.830 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:56 smithi122 ceph-mon[27145]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:47:56.830 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:56 smithi122 ceph-mon[27145]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:47:56.830 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:56 smithi122 ceph-mon[27145]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:47:56.830 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:56 smithi122 ceph-mon[27145]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:47:56.830 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:56 smithi122 ceph-mon[27145]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:47:56.830 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:56 smithi122 ceph-mon[27145]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:47:56.830 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:56 smithi122 ceph-mon[27145]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:47:56.830 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:56 smithi122 ceph-mon[27145]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-02T04:47:56.830 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:56 smithi122 ceph-mon[27145]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-02T04:47:56.830 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:56 smithi122 ceph-mon[27145]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-02T04:47:56.830 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:56 smithi122 ceph-mon[27145]: Manager daemon smithi053.legdmv is now available 2024-04-02T04:47:57.319 INFO:teuthology.orchestra.run.smithi053.stdout: 5 150436 11.40 MB/sec execute 187 sec latency 56.741 ms 2024-04-02T04:47:57.669 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:57 smithi053 ceph-mon[67180]: failed to encode map e72 with expected crc 2024-04-02T04:47:57.669 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:57 smithi053 ceph-mon[67180]: failed to encode map e72 with expected crc 2024-04-02T04:47:57.669 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:57 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:47:57.669 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:57 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:47:57.669 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:57 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:47:57.670 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:57 smithi053 ceph-mon[67180]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:47:57.670 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:57 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/trash_purge_schedule"}]: dispatch 2024-04-02T04:47:57.670 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:57 smithi053 ceph-mon[67180]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/trash_purge_schedule"}]: dispatch 2024-04-02T04:47:57.670 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:57 smithi053 ceph-mon[67180]: [02/Apr/2024:04:47:57] ENGINE Bus STARTING 2024-04-02T04:47:57.670 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:57 smithi053 ceph-mon[67180]: [02/Apr/2024:04:47:57] ENGINE Serving on https://172.21.15.53:7150 2024-04-02T04:47:57.670 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:57 smithi053 ceph-mon[67180]: [02/Apr/2024:04:47:57] ENGINE Serving on http://172.21.15.53:8765 2024-04-02T04:47:57.670 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:57 smithi053 ceph-mon[67180]: [02/Apr/2024:04:47:57] ENGINE Bus STARTED 2024-04-02T04:47:57.670 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:57 smithi053 ceph-mon[67180]: mgrmap e38: smithi053.legdmv(active, since 1.01583s), standbys: smithi122.cpwlyk 2024-04-02T04:47:57.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:57 smithi122 ceph-mon[27145]: failed to encode map e72 with expected crc 2024-04-02T04:47:57.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:57 smithi122 ceph-mon[27145]: failed to encode map e72 with expected crc 2024-04-02T04:47:57.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:57 smithi122 ceph-mon[27145]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:47:57.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:57 smithi122 ceph-mon[27145]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:47:57.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:57 smithi122 ceph-mon[27145]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:47:57.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:57 smithi122 ceph-mon[27145]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/mirror_snapshot_schedule"}]: dispatch 2024-04-02T04:47:57.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:57 smithi122 ceph-mon[27145]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/trash_purge_schedule"}]: dispatch 2024-04-02T04:47:57.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:57 smithi122 ceph-mon[27145]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi053.legdmv/trash_purge_schedule"}]: dispatch 2024-04-02T04:47:57.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:57 smithi122 ceph-mon[27145]: [02/Apr/2024:04:47:57] ENGINE Bus STARTING 2024-04-02T04:47:57.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:57 smithi122 ceph-mon[27145]: [02/Apr/2024:04:47:57] ENGINE Serving on https://172.21.15.53:7150 2024-04-02T04:47:57.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:57 smithi122 ceph-mon[27145]: [02/Apr/2024:04:47:57] ENGINE Serving on http://172.21.15.53:8765 2024-04-02T04:47:57.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:57 smithi122 ceph-mon[27145]: [02/Apr/2024:04:47:57] ENGINE Bus STARTED 2024-04-02T04:47:57.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:57 smithi122 ceph-mon[27145]: mgrmap e38: smithi053.legdmv(active, since 1.01583s), standbys: smithi122.cpwlyk 2024-04-02T04:47:58.320 INFO:teuthology.orchestra.run.smithi053.stdout: 5 150740 11.38 MB/sec execute 188 sec latency 60.432 ms 2024-04-02T04:47:58.733 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:58 smithi053 ceph-mon[67180]: pgmap v3: 97 pgs: 97 active+clean; 328 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-04-02T04:47:58.733 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:58 smithi053 ceph-mon[67180]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-02T04:47:58.733 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:58 smithi053 ceph-mon[67180]: Cluster is now healthy 2024-04-02T04:47:58.876 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:58 smithi122 ceph-mon[27145]: pgmap v3: 97 pgs: 97 active+clean; 328 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-04-02T04:47:58.876 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:58 smithi122 ceph-mon[27145]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-02T04:47:58.876 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:58 smithi122 ceph-mon[27145]: Cluster is now healthy 2024-04-02T04:47:59.320 INFO:teuthology.orchestra.run.smithi053.stdout: 5 151110 11.36 MB/sec execute 189 sec latency 57.667 ms 2024-04-02T04:47:59.749 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:59 smithi053 ceph-mon[67180]: pgmap v4: 97 pgs: 97 active+clean; 328 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-04-02T04:47:59.750 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:47:59 smithi053 ceph-mon[67180]: mgrmap e39: smithi053.legdmv(active, since 2s), standbys: smithi122.cpwlyk 2024-04-02T04:47:59.763 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:59 smithi122 ceph-mon[27145]: pgmap v4: 97 pgs: 97 active+clean; 328 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-04-02T04:47:59.764 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:47:59 smithi122 ceph-mon[27145]: mgrmap e39: smithi053.legdmv(active, since 2s), standbys: smithi122.cpwlyk 2024-04-02T04:48:00.320 INFO:teuthology.orchestra.run.smithi053.stdout: 5 151418 11.35 MB/sec execute 190 sec latency 66.567 ms 2024-04-02T04:48:01.320 INFO:teuthology.orchestra.run.smithi053.stdout: 5 151873 11.39 MB/sec execute 191 sec latency 37.305 ms 2024-04-02T04:48:01.795 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:01 smithi053 ceph-mon[67180]: pgmap v5: 97 pgs: 97 active+clean; 328 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-04-02T04:48:01.795 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:01 smithi053 ceph-mon[67180]: mgrmap e40: smithi053.legdmv(active, since 4s), standbys: smithi122.cpwlyk 2024-04-02T04:48:01.796 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:01 smithi053 ceph-mon[67180]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:01.796 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:01 smithi053 ceph-mon[67180]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:01.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:01 smithi122 ceph-mon[27145]: pgmap v5: 97 pgs: 97 active+clean; 328 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-04-02T04:48:01.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:01 smithi122 ceph-mon[27145]: mgrmap e40: smithi053.legdmv(active, since 4s), standbys: smithi122.cpwlyk 2024-04-02T04:48:01.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:01 smithi122 ceph-mon[27145]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:01.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:01 smithi122 ceph-mon[27145]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:02.320 INFO:teuthology.orchestra.run.smithi053.stdout: 5 152344 11.40 MB/sec execute 192 sec latency 31.627 ms 2024-04-02T04:48:03.320 INFO:teuthology.orchestra.run.smithi053.stdout: 5 152809 11.41 MB/sec execute 193 sec latency 25.461 ms 2024-04-02T04:48:03.559 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:03 smithi053 ceph-mon[67180]: pgmap v6: 97 pgs: 97 active+clean; 332 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 41 KiB/s rd, 2.9 MiB/s wr, 407 op/s 2024-04-02T04:48:03.560 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:03 smithi053 ceph-mon[67180]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:03.560 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:03 smithi053 ceph-mon[67180]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:03.590 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:03 smithi122 ceph-mon[27145]: pgmap v6: 97 pgs: 97 active+clean; 332 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 41 KiB/s rd, 2.9 MiB/s wr, 407 op/s 2024-04-02T04:48:03.590 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:03 smithi122 ceph-mon[27145]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:03.590 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:03 smithi122 ceph-mon[27145]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:04.320 INFO:teuthology.orchestra.run.smithi053.stdout: 5 153225 11.39 MB/sec execute 194 sec latency 39.919 ms 2024-04-02T04:48:05.320 INFO:teuthology.orchestra.run.smithi053.stdout: 5 153671 11.40 MB/sec execute 195 sec latency 45.361 ms 2024-04-02T04:48:06.321 INFO:teuthology.orchestra.run.smithi053.stdout: 5 154067 11.38 MB/sec execute 196 sec latency 65.501 ms 2024-04-02T04:48:06.350 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:06 smithi053 ceph-mon[67180]: pgmap v7: 97 pgs: 97 active+clean; 333 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 33 KiB/s rd, 4.2 MiB/s wr, 390 op/s 2024-04-02T04:48:06.351 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:06 smithi053 ceph-mon[67180]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:06.351 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:06 smithi053 ceph-mon[67180]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:06.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:06 smithi122 ceph-mon[27145]: pgmap v7: 97 pgs: 97 active+clean; 333 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 33 KiB/s rd, 4.2 MiB/s wr, 390 op/s 2024-04-02T04:48:06.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:06 smithi122 ceph-mon[27145]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:06.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:06 smithi122 ceph-mon[27145]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:07.320 INFO:teuthology.orchestra.run.smithi053.stdout: 5 154408 11.35 MB/sec execute 197 sec latency 42.583 ms 2024-04-02T04:48:07.614 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:07 smithi053 ceph-mon[67180]: pgmap v8: 97 pgs: 97 active+clean; 333 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 3.2 MiB/s wr, 303 op/s 2024-04-02T04:48:07.614 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:07 smithi053 ceph-mon[67180]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:07.614 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:07 smithi053 ceph-mon[67180]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:07.614 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:07 smithi053 ceph-mon[67180]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:07.614 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:07 smithi053 ceph-mon[67180]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:07.614 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:07 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:07.614 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:07 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:48:07.614 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:07 smithi053 ceph-mon[67180]: Updating smithi053:/etc/ceph/ceph.conf 2024-04-02T04:48:07.614 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:07 smithi053 ceph-mon[67180]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-02T04:48:07.614 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:07 smithi053 ceph-mon[67180]: Updating smithi122:/var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T04:48:07.615 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:07 smithi053 ceph-mon[67180]: Updating smithi053:/var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T04:48:07.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:07 smithi122 ceph-mon[27145]: pgmap v8: 97 pgs: 97 active+clean; 333 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 3.2 MiB/s wr, 303 op/s 2024-04-02T04:48:07.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:07 smithi122 ceph-mon[27145]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:07.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:07 smithi122 ceph-mon[27145]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:07.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:07 smithi122 ceph-mon[27145]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:07.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:07 smithi122 ceph-mon[27145]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:07.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:07 smithi122 ceph-mon[27145]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:07.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:07 smithi122 ceph-mon[27145]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:48:07.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:07 smithi122 ceph-mon[27145]: Updating smithi053:/etc/ceph/ceph.conf 2024-04-02T04:48:07.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:07 smithi122 ceph-mon[27145]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-02T04:48:07.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:07 smithi122 ceph-mon[27145]: Updating smithi122:/var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T04:48:07.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:07 smithi122 ceph-mon[27145]: Updating smithi053:/var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T04:48:08.320 INFO:teuthology.orchestra.run.smithi053.stdout: 5 154806 11.34 MB/sec execute 198 sec latency 39.414 ms 2024-04-02T04:48:09.320 INFO:teuthology.orchestra.run.smithi053.stdout: 5 155510 11.44 MB/sec execute 199 sec latency 25.373 ms 2024-04-02T04:48:09.340 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:09 smithi053 ceph-mon[67180]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-04-02T04:48:09.340 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:09 smithi053 ceph-mon[67180]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-02T04:48:09.340 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:09 smithi053 ceph-mon[67180]: Updating smithi053:/var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T04:48:09.340 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:09 smithi053 ceph-mon[67180]: Updating smithi122:/var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T04:48:09.340 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:09 smithi053 ceph-mon[67180]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:09.340 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:09 smithi053 ceph-mon[67180]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:09.340 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:09 smithi053 ceph-mon[67180]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:09.340 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:09 smithi053 ceph-mon[67180]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:09.340 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:09 smithi053 ceph-mon[67180]: pgmap v9: 97 pgs: 97 active+clean; 327 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 5.2 MiB/s wr, 548 op/s 2024-04-02T04:48:09.340 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:09 smithi053 ceph-mon[67180]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:09.340 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:09 smithi053 ceph-mon[67180]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:09.340 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:09 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:48:09.340 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:09 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:48:09.341 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:09 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-02T04:48:09.420 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:09 smithi122 ceph-mon[27145]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-04-02T04:48:09.420 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:09 smithi122 ceph-mon[27145]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-02T04:48:09.420 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:09 smithi122 ceph-mon[27145]: Updating smithi053:/var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T04:48:09.420 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:09 smithi122 ceph-mon[27145]: Updating smithi122:/var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T04:48:09.420 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:09 smithi122 ceph-mon[27145]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:09.420 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:09 smithi122 ceph-mon[27145]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:09.420 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:09 smithi122 ceph-mon[27145]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:09.420 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:09 smithi122 ceph-mon[27145]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:09.420 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:09 smithi122 ceph-mon[27145]: pgmap v9: 97 pgs: 97 active+clean; 327 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 5.2 MiB/s wr, 548 op/s 2024-04-02T04:48:09.421 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:09 smithi122 ceph-mon[27145]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:09.421 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:09 smithi122 ceph-mon[27145]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:09.421 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:09 smithi122 ceph-mon[27145]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:48:09.421 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:09 smithi122 ceph-mon[27145]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:48:09.421 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:09 smithi122 ceph-mon[27145]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-02T04:48:10.320 INFO:teuthology.orchestra.run.smithi053.stdout: 5 156184 11.46 MB/sec execute 200 sec latency 25.543 ms 2024-04-02T04:48:10.386 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:10 smithi053 ceph-mon[67180]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-02T04:48:10.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:10 smithi122 ceph-mon[27145]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-02T04:48:11.320 INFO:teuthology.orchestra.run.smithi053.stdout: 5 156526 11.45 MB/sec execute 201 sec latency 31.272 ms 2024-04-02T04:48:11.391 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:11 smithi053 ceph-mon[67180]: pgmap v10: 97 pgs: 97 active+clean; 327 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 4.8 MiB/s wr, 501 op/s 2024-04-02T04:48:11.392 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:11 smithi053 ceph-mon[67180]: Upgrade: Updating mon.smithi122 2024-04-02T04:48:11.392 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:11 smithi053 ceph-mon[67180]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:11.392 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:11 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T04:48:11.392 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:11 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-02T04:48:11.392 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:11 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:11.392 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:11 smithi053 ceph-mon[67180]: Deploying daemon mon.smithi122 on smithi122 2024-04-02T04:48:11.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:11 smithi122 ceph-mon[27145]: pgmap v10: 97 pgs: 97 active+clean; 327 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 4.8 MiB/s wr, 501 op/s 2024-04-02T04:48:11.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:11 smithi122 ceph-mon[27145]: Upgrade: Updating mon.smithi122 2024-04-02T04:48:11.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:11 smithi122 ceph-mon[27145]: from='mgr.34104 ' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:11.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:11 smithi122 ceph-mon[27145]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T04:48:11.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:11 smithi122 ceph-mon[27145]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-02T04:48:11.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:11 smithi122 ceph-mon[27145]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:11.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:11 smithi122 ceph-mon[27145]: Deploying daemon mon.smithi122 on smithi122 2024-04-02T04:48:12.320 INFO:teuthology.orchestra.run.smithi053.stdout: 5 156848 11.42 MB/sec execute 202 sec latency 43.370 ms 2024-04-02T04:48:12.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:12 smithi122 ceph-mon[27145]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:48:12.533 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:12 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:48:13.055 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:12 smithi122 systemd[1]: Stopping Ceph mon.smithi122 for 48e16742-f0aa-11ee-b647-cb9ed24678a4... 2024-04-02T04:48:13.320 INFO:teuthology.orchestra.run.smithi053.stdout: 5 157321 11.43 MB/sec execute 203 sec latency 36.140 ms 2024-04-02T04:48:13.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:13 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi122[27128]: 2024-04-02T04:48:13.050+0000 7f0727121700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi122 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T04:48:13.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:13 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi122[27128]: 2024-04-02T04:48:13.050+0000 7f0727121700 -1 mon.smithi122@1(peon) e2 *** Got Signal Terminated *** 2024-04-02T04:48:13.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:13 smithi122 podman[53052]: 2024-04-02 04:48:13.122427165 +0000 UTC m=+0.176841659 container died a2df07e0883da78b0bba796d0c5e30ec83ed259f2ca265d109157f5f93a81c6c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi122, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, vendor=Red Hat, Inc., io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, name=centos-stream, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, version=8, release=754, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8) 2024-04-02T04:48:13.756 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:13 smithi122 podman[53052]: 2024-04-02 04:48:13.518202965 +0000 UTC m=+0.572617468 container cleanup a2df07e0883da78b0bba796d0c5e30ec83ed259f2ca265d109157f5f93a81c6c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi122, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, name=centos-stream, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, ceph=True, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, io.buildah.version=1.19.8, release=754, io.openshift.tags=base centos centos-stream, version=8) 2024-04-02T04:48:13.756 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:13 smithi122 bash[53052]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi122 2024-04-02T04:48:14.032 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:13 smithi122 podman[53064]: 2024-04-02 04:48:13.752877898 +0000 UTC m=+0.632277623 container remove a2df07e0883da78b0bba796d0c5e30ec83ed259f2ca265d109157f5f93a81c6c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi122, com.redhat.component=centos-stream-container, GIT_CLEAN=True, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, name=centos-stream, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, build-date=2022-05-03T08:36:31.336870) 2024-04-02T04:48:14.320 INFO:teuthology.orchestra.run.smithi053.stdout: 5 157899 11.42 MB/sec execute 204 sec latency 33.364 ms 2024-04-02T04:48:14.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 systemd[1]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@mon.smithi122.service: Deactivated successfully. 2024-04-02T04:48:14.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 systemd[1]: Stopped Ceph mon.smithi122 for 48e16742-f0aa-11ee-b647-cb9ed24678a4. 2024-04-02T04:48:14.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 systemd[1]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@mon.smithi122.service: Consumed 7.063s CPU time. 2024-04-02T04:48:14.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 systemd[1]: Starting Ceph mon.smithi122 for 48e16742-f0aa-11ee-b647-cb9ed24678a4... 2024-04-02T04:48:14.777 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 podman[53192]: 2024-04-02 04:48:14.41404794 +0000 UTC m=+0.020084560 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:48:14.778 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 podman[53192]: 2024-04-02 04:48:14.528898117 +0000 UTC m=+0.134934727 container create 44b200fed3c563ba8f5fbd9ed012b50611c7f0fa40150d8f77b52c5e4dbca5cf (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi122, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-6cec793, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-02T04:48:15.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 podman[53192]: 2024-04-02 04:48:14.77448099 +0000 UTC m=+0.380517640 container init 44b200fed3c563ba8f5fbd9ed012b50611c7f0fa40150d8f77b52c5e4dbca5cf (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi122, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, RELEASE=squid-6cec793, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_CLEAN=True) 2024-04-02T04:48:15.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 podman[53192]: 2024-04-02 04:48:14.779072983 +0000 UTC m=+0.385109594 container start 44b200fed3c563ba8f5fbd9ed012b50611c7f0fa40150d8f77b52c5e4dbca5cf (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi122, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, ceph=True, RELEASE=squid-6cec793, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T04:48:15.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: set uid:gid to 167:167 (ceph:ceph) 2024-04-02T04:48:15.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev), process ceph-mon, pid 2 2024-04-02T04:48:15.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: pidfile_write: ignore empty --pid-file 2024-04-02T04:48:15.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: load: jerasure load: lrc 2024-04-02T04:48:15.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: RocksDB version: 7.9.2 2024-04-02T04:48:15.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Git sha 0 2024-04-02T04:48:15.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Compile date 2024-04-01 15:23:26 2024-04-02T04:48:15.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: DB SUMMARY 2024-04-02T04:48:15.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: DB Session ID: BQQPA2PH86Q20LUQWL92 2024-04-02T04:48:15.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: CURRENT file: CURRENT 2024-04-02T04:48:15.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: IDENTITY file: IDENTITY 2024-04-02T04:48:15.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: MANIFEST file: MANIFEST-000009 size: 903 Bytes 2024-04-02T04:48:15.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi122/store.db dir, Total Num: 1, files: 000024.sst 2024-04-02T04:48:15.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi122/store.db: 000022.log size: 7323696 ; 2024-04-02T04:48:15.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.error_if_exists: 0 2024-04-02T04:48:15.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.create_if_missing: 0 2024-04-02T04:48:15.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.paranoid_checks: 1 2024-04-02T04:48:15.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-02T04:48:15.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-02T04:48:15.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-02T04:48:15.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.env: 0x5566a3071c60 2024-04-02T04:48:15.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.fs: PosixFileSystem 2024-04-02T04:48:15.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.info_log: 0x5566a4ed99a0 2024-04-02T04:48:15.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.max_file_opening_threads: 16 2024-04-02T04:48:15.148 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.statistics: (nil) 2024-04-02T04:48:15.148 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.use_fsync: 0 2024-04-02T04:48:15.149 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.max_log_file_size: 0 2024-04-02T04:48:15.149 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-02T04:48:15.149 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-02T04:48:15.149 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.keep_log_file_num: 1000 2024-04-02T04:48:15.149 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.recycle_log_file_num: 0 2024-04-02T04:48:15.149 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.allow_fallocate: 1 2024-04-02T04:48:15.149 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.allow_mmap_reads: 0 2024-04-02T04:48:15.149 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.allow_mmap_writes: 0 2024-04-02T04:48:15.149 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.use_direct_reads: 0 2024-04-02T04:48:15.149 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-02T04:48:15.149 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.create_missing_column_families: 0 2024-04-02T04:48:15.149 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.db_log_dir: 2024-04-02T04:48:15.149 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.wal_dir: 2024-04-02T04:48:15.149 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-02T04:48:15.149 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-02T04:48:15.150 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-02T04:48:15.150 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-02T04:48:15.150 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-02T04:48:15.150 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-02T04:48:15.150 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.advise_random_on_open: 1 2024-04-02T04:48:15.150 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.db_write_buffer_size: 0 2024-04-02T04:48:15.150 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.write_buffer_manager: 0x5566a4eac1e0 2024-04-02T04:48:15.150 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-02T04:48:15.150 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-02T04:48:15.150 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-02T04:48:15.150 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.rate_limiter: (nil) 2024-04-02T04:48:15.150 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-02T04:48:15.150 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.wal_recovery_mode: 2 2024-04-02T04:48:15.150 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.enable_thread_tracking: 0 2024-04-02T04:48:15.150 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.enable_pipelined_write: 0 2024-04-02T04:48:15.151 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.unordered_write: 0 2024-04-02T04:48:15.151 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-02T04:48:15.151 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-02T04:48:15.151 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-02T04:48:15.151 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-02T04:48:15.151 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.row_cache: None 2024-04-02T04:48:15.151 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.wal_filter: None 2024-04-02T04:48:15.151 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-02T04:48:15.151 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.allow_ingest_behind: 0 2024-04-02T04:48:15.151 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.two_write_queues: 0 2024-04-02T04:48:15.151 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.manual_wal_flush: 0 2024-04-02T04:48:15.151 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.wal_compression: 0 2024-04-02T04:48:15.151 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.atomic_flush: 0 2024-04-02T04:48:15.151 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-02T04:48:15.151 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-02T04:48:15.152 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-02T04:48:15.152 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.log_readahead_size: 0 2024-04-02T04:48:15.152 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-02T04:48:15.152 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.best_efforts_recovery: 0 2024-04-02T04:48:15.152 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-02T04:48:15.152 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-02T04:48:15.152 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.allow_data_in_errors: 0 2024-04-02T04:48:15.152 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.db_host_id: __hostname__ 2024-04-02T04:48:15.152 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-02T04:48:15.152 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.max_background_jobs: 2 2024-04-02T04:48:15.152 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.max_background_compactions: -1 2024-04-02T04:48:15.152 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.max_subcompactions: 1 2024-04-02T04:48:15.152 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-02T04:48:15.152 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-02T04:48:15.152 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-02T04:48:15.152 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.max_total_wal_size: 0 2024-04-02T04:48:15.153 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-02T04:48:15.153 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-02T04:48:15.153 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-02T04:48:15.153 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-02T04:48:15.153 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.max_open_files: -1 2024-04-02T04:48:15.153 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.bytes_per_sync: 0 2024-04-02T04:48:15.153 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-02T04:48:15.153 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-02T04:48:15.153 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.compaction_readahead_size: 0 2024-04-02T04:48:15.153 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.max_background_flushes: -1 2024-04-02T04:48:15.153 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Compression algorithms supported: 2024-04-02T04:48:15.153 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-02T04:48:15.153 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: kZSTD supported: 0 2024-04-02T04:48:15.153 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: kXpressCompression supported: 0 2024-04-02T04:48:15.153 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: kLZ4HCCompression supported: 1 2024-04-02T04:48:15.153 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: kZlibCompression supported: 1 2024-04-02T04:48:15.154 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: kSnappyCompression supported: 1 2024-04-02T04:48:15.154 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: kLZ4Compression supported: 1 2024-04-02T04:48:15.154 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: kBZip2Compression supported: 0 2024-04-02T04:48:15.154 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-02T04:48:15.154 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-02T04:48:15.154 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi122/store.db/MANIFEST-000009 2024-04-02T04:48:15.154 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-02T04:48:15.154 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-02T04:48:15.154 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.merge_operator: 2024-04-02T04:48:15.154 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.compaction_filter: None 2024-04-02T04:48:15.154 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.compaction_filter_factory: None 2024-04-02T04:48:15.154 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.sst_partitioner_factory: None 2024-04-02T04:48:15.154 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-02T04:48:15.154 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-02T04:48:15.154 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5566a4ed9aa0) 2024-04-02T04:48:15.155 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: cache_index_and_filter_blocks: 1 2024-04-02T04:48:15.155 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-02T04:48:15.155 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-02T04:48:15.155 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: pin_top_level_index_and_filter: 1 2024-04-02T04:48:15.155 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: index_type: 0 2024-04-02T04:48:15.155 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: data_block_index_type: 0 2024-04-02T04:48:15.155 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: index_shortening: 1 2024-04-02T04:48:15.155 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-02T04:48:15.155 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: checksum: 4 2024-04-02T04:48:15.155 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: no_block_cache: 0 2024-04-02T04:48:15.155 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: block_cache: 0x5566a4edd090 2024-04-02T04:48:15.155 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: block_cache_name: BinnedLRUCache 2024-04-02T04:48:15.155 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: block_cache_options: 2024-04-02T04:48:15.155 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: capacity : 536870912 2024-04-02T04:48:15.155 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: num_shard_bits : 4 2024-04-02T04:48:15.155 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: strict_capacity_limit : 0 2024-04-02T04:48:15.156 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: high_pri_pool_ratio: 0.000 2024-04-02T04:48:15.156 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: block_cache_compressed: (nil) 2024-04-02T04:48:15.156 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: persistent_cache: (nil) 2024-04-02T04:48:15.156 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: block_size: 4096 2024-04-02T04:48:15.156 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: block_size_deviation: 10 2024-04-02T04:48:15.156 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: block_restart_interval: 16 2024-04-02T04:48:15.156 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: index_block_restart_interval: 1 2024-04-02T04:48:15.156 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: metadata_block_size: 4096 2024-04-02T04:48:15.156 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: partition_filters: 0 2024-04-02T04:48:15.156 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: use_delta_encoding: 1 2024-04-02T04:48:15.156 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: filter_policy: bloomfilter 2024-04-02T04:48:15.156 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: whole_key_filtering: 1 2024-04-02T04:48:15.156 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: verify_compression: 0 2024-04-02T04:48:15.156 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: read_amp_bytes_per_bit: 0 2024-04-02T04:48:15.156 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: format_version: 5 2024-04-02T04:48:15.156 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: enable_index_compression: 1 2024-04-02T04:48:15.157 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: block_align: 0 2024-04-02T04:48:15.157 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: max_auto_readahead_size: 262144 2024-04-02T04:48:15.157 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: prepopulate_block_cache: 0 2024-04-02T04:48:15.157 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: initial_auto_readahead_size: 8192 2024-04-02T04:48:15.157 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: num_file_reads_for_auto_readahead: 2 2024-04-02T04:48:15.157 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.write_buffer_size: 33554432 2024-04-02T04:48:15.157 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.max_write_buffer_number: 2 2024-04-02T04:48:15.157 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.compression: NoCompression 2024-04-02T04:48:15.157 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.bottommost_compression: Disabled 2024-04-02T04:48:15.157 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.prefix_extractor: nullptr 2024-04-02T04:48:15.157 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-02T04:48:15.157 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.num_levels: 7 2024-04-02T04:48:15.157 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-02T04:48:15.157 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-02T04:48:15.157 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-02T04:48:15.157 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-02T04:48:15.158 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-02T04:48:15.158 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-02T04:48:15.158 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-02T04:48:15.158 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-02T04:48:15.158 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-02T04:48:15.158 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-02T04:48:15.158 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-02T04:48:15.158 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-02T04:48:15.158 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-02T04:48:15.158 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.compression_opts.level: 32767 2024-04-02T04:48:15.158 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.compression_opts.strategy: 0 2024-04-02T04:48:15.158 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-02T04:48:15.158 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-02T04:48:15.158 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-02T04:48:15.158 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-02T04:48:15.159 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.compression_opts.enabled: false 2024-04-02T04:48:15.159 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-02T04:48:15.159 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-02T04:48:15.159 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-02T04:48:15.159 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-02T04:48:15.159 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.target_file_size_base: 67108864 2024-04-02T04:48:15.159 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-02T04:48:15.159 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-02T04:48:15.159 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-02T04:48:15.159 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-02T04:48:15.159 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-02T04:48:15.159 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-02T04:48:15.159 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-02T04:48:15.159 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-02T04:48:15.160 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-02T04:48:15.160 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-02T04:48:15.160 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-02T04:48:15.160 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-02T04:48:15.160 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-02T04:48:15.160 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-02T04:48:15.160 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.arena_block_size: 1048576 2024-04-02T04:48:15.160 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-02T04:48:15.160 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-02T04:48:15.160 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.disable_auto_compactions: 0 2024-04-02T04:48:15.160 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-02T04:48:15.160 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-02T04:48:15.160 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-02T04:48:15.160 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-02T04:48:15.160 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-02T04:48:15.161 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-02T04:48:15.161 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-02T04:48:15.161 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-02T04:48:15.161 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-02T04:48:15.161 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-02T04:48:15.161 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.table_properties_collectors: 2024-04-02T04:48:15.161 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.inplace_update_support: 0 2024-04-02T04:48:15.161 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-02T04:48:15.161 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-02T04:48:15.161 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-02T04:48:15.161 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-02T04:48:15.161 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.bloom_locality: 0 2024-04-02T04:48:15.161 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.max_successive_merges: 0 2024-04-02T04:48:15.161 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-02T04:48:15.162 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.paranoid_file_checks: 0 2024-04-02T04:48:15.162 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.force_consistency_checks: 1 2024-04-02T04:48:15.162 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.report_bg_io_stats: 0 2024-04-02T04:48:15.162 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.ttl: 2592000 2024-04-02T04:48:15.162 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-02T04:48:15.162 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-02T04:48:15.162 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-02T04:48:15.162 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.enable_blob_files: false 2024-04-02T04:48:15.162 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.min_blob_size: 0 2024-04-02T04:48:15.162 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.blob_file_size: 268435456 2024-04-02T04:48:15.162 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-02T04:48:15.162 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-02T04:48:15.162 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-02T04:48:15.163 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-02T04:48:15.163 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-02T04:48:15.163 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.blob_file_starting_level: 0 2024-04-02T04:48:15.163 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-02T04:48:15.163 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 24.sst 2024-04-02T04:48:15.163 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-02T04:48:15.163 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi122/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 26, last_sequence is 12197, log_number is 22,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-02T04:48:15.163 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 22 2024-04-02T04:48:15.163 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d1f8c5ac-ae9d-4986-9636-059ae4518e75 2024-04-02T04:48:15.163 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712033294818982, "job": 1, "event": "recovery_started", "wal_files": [22]} 2024-04-02T04:48:15.163 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #22 mode 2 2024-04-02T04:48:15.163 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712033294845699, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 27, "file_size": 4442570, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12202, "largest_seqno": 14109, "table_properties": {"data_size": 4434305, "index_size": 5325, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1925, "raw_key_size": 17192, "raw_average_key_size": 23, "raw_value_size": 4418102, "raw_average_value_size": 5922, "num_data_blocks": 242, "num_entries": 746, "num_filter_entries": 746, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712033294, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d1f8c5ac-ae9d-4986-9636-059ae4518e75", "db_session_id": "BQQPA2PH86Q20LUQWL92", "orig_file_number": 27, "seqno_to_time_mapping": "N/A"}} 2024-04-02T04:48:15.163 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712033294845868, "job": 1, "event": "recovery_finished"} 2024-04-02T04:48:15.163 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: [db/version_set.cc:5047] Creating manifest 29 2024-04-02T04:48:15.164 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-02T04:48:15.164 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi122/store.db/000022.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-02T04:48:15.164 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5566a4fca000 2024-04-02T04:48:15.164 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: DB pointer 0x5566a4fb8000 2024-04-02T04:48:15.164 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-02T04:48:15.164 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-02T04:48:15.164 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: ** DB Stats ** 2024-04-02T04:48:15.164 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-02T04:48:15.164 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-02T04:48:15.164 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-02T04:48:15.164 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-02T04:48:15.164 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-02T04:48:15.164 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-02T04:48:15.165 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-02T04:48:15.165 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-04-02T04:48:15.165 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: ** Compaction Stats [default] ** 2024-04-02T04:48:15.165 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-02T04:48:15.165 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-02T04:48:15.165 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: L0 1/0 4.24 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 384.0 0.01 0.00 1 0.011 0 0 0.0 0.0 2024-04-02T04:48:15.165 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: L6 1/0 6.45 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-02T04:48:15.165 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Sum 2/0 10.68 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 384.0 0.01 0.00 1 0.011 0 0 0.0 0.0 2024-04-02T04:48:15.165 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 384.0 0.01 0.00 1 0.011 0 0 0.0 0.0 2024-04-02T04:48:15.165 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-04-02T04:48:15.165 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: ** Compaction Stats [default] ** 2024-04-02T04:48:15.165 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-02T04:48:15.165 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-02T04:48:15.165 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 384.0 0.01 0.00 1 0.011 0 0 0.0 0.0 2024-04-02T04:48:15.166 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-04-02T04:48:15.166 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-02T04:48:15.166 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-04-02T04:48:15.166 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-02T04:48:15.166 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Flush(GB): cumulative 0.004, interval 0.004 2024-04-02T04:48:15.166 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-02T04:48:15.166 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-02T04:48:15.166 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-02T04:48:15.166 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-02T04:48:15.166 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Cumulative compaction: 0.00 GB write, 123.14 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-02T04:48:15.166 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Interval compaction: 0.00 GB write, 123.14 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-02T04:48:15.166 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-02T04:48:15.167 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Block cache BinnedLRUCache@0x5566a4edd090#2 capacity: 512.00 MB usage: 8.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.5e-05 secs_since: 0 2024-04-02T04:48:15.167 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.03 KB,0.00038743%) IndexBlock(1,6.08 KB,0.00115931%) Misc(1,0.00 KB,0%) 2024-04-02T04:48:15.167 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-04-02T04:48:15.167 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-02T04:48:15.167 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: starting mon.smithi122 rank 1 at public addrs [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] at bind addrs [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi122 fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 2024-04-02T04:48:15.167 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: mon.smithi122@-1(???) e2 preinit fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 2024-04-02T04:48:15.167 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: mon.smithi122@-1(???).mds e7 new map 2024-04-02T04:48:15.167 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: mon.smithi122@-1(???).mds e7 print_map 2024-04-02T04:48:15.167 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: e7 2024-04-02T04:48:15.167 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-02T04:48:15.167 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-02T04:48:15.167 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: legacy client fscid: 1 2024-04-02T04:48:15.167 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-04-02T04:48:15.167 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Filesystem 'foofs' (1) 2024-04-02T04:48:15.168 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: fs_name foofs 2024-04-02T04:48:15.168 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: epoch 5 2024-04-02T04:48:15.168 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-02T04:48:15.168 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: created 2024-04-02T04:42:12.430527+0000 2024-04-02T04:48:15.168 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: modified 2024-04-02T04:42:18.452666+0000 2024-04-02T04:48:15.168 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: tableserver 0 2024-04-02T04:48:15.168 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: root 0 2024-04-02T04:48:15.168 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: session_timeout 60 2024-04-02T04:48:15.168 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: session_autoclose 300 2024-04-02T04:48:15.168 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: max_file_size 1099511627776 2024-04-02T04:48:15.168 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: max_xattr_size 65536 2024-04-02T04:48:15.168 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: required_client_features {} 2024-04-02T04:48:15.168 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: last_failure 0 2024-04-02T04:48:15.168 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: last_failure_osd_epoch 0 2024-04-02T04:48:15.169 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-02T04:48:15.169 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: max_mds 1 2024-04-02T04:48:15.169 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: in 0 2024-04-02T04:48:15.169 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: up {0=14478} 2024-04-02T04:48:15.169 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: failed 2024-04-02T04:48:15.169 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: damaged 2024-04-02T04:48:15.169 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: stopped 2024-04-02T04:48:15.169 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: data_pools [3] 2024-04-02T04:48:15.169 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: metadata_pool 2 2024-04-02T04:48:15.169 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: inline_data disabled 2024-04-02T04:48:15.169 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: balancer 2024-04-02T04:48:15.169 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: bal_rank_mask -1 2024-04-02T04:48:15.169 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: standby_count_wanted 1 2024-04-02T04:48:15.169 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: qdb_cluster leader: 0 members: 2024-04-02T04:48:15.170 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: [mds.foofs.smithi053.nfrfot{0:14478} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.53:6834/2480061460,v1:172.21.15.53:6835/2480061460] compat {c=[1],r=[1],i=[7ff]}] 2024-04-02T04:48:15.170 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-04-02T04:48:15.170 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-04-02T04:48:15.170 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Standby daemons: 2024-04-02T04:48:15.170 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-04-02T04:48:15.170 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: [mds.foofs.smithi122.mwoyoh{-1:24301} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6832/233414887,v1:172.21.15.122:6833/233414887] compat {c=[1],r=[1],i=[7ff]}] 2024-04-02T04:48:15.170 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: mon.smithi122@-1(???).osd e72 crush map has features 3314933000854323200, adjusting msgr requires 2024-04-02T04:48:15.170 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: mon.smithi122@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-04-02T04:48:15.170 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: mon.smithi122@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-04-02T04:48:15.170 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: mon.smithi122@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-04-02T04:48:15.170 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 ceph-mon[53211]: mon.smithi122@-1(???).paxosservice(auth 1..30) refresh upgraded, format 0 -> 3 2024-04-02T04:48:15.170 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 bash[53192]: 44b200fed3c563ba8f5fbd9ed012b50611c7f0fa40150d8f77b52c5e4dbca5cf 2024-04-02T04:48:15.170 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:14 smithi122 systemd[1]: Started Ceph mon.smithi122 for 48e16742-f0aa-11ee-b647-cb9ed24678a4. 2024-04-02T04:48:15.320 INFO:teuthology.orchestra.run.smithi053.stdout: 5 158410 11.43 MB/sec execute 205 sec latency 33.604 ms 2024-04-02T04:48:16.320 INFO:teuthology.orchestra.run.smithi053.stdout: 5 158920 11.47 MB/sec execute 206 sec latency 30.772 ms 2024-04-02T04:48:16.404 INFO:teuthology.orchestra.run.smithi053.stdout:true 2024-04-02T04:48:16.569 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:16 smithi053 ceph-mon[67180]: pgmap v11: 97 pgs: 97 active+clean; 327 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 4.8 MiB/s wr, 501 op/s 2024-04-02T04:48:16.570 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:16 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi053"}]: dispatch 2024-04-02T04:48:16.570 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:16 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-02T04:48:16.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:16 smithi122 ceph-mon[53211]: pgmap v11: 97 pgs: 97 active+clean; 327 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 4.8 MiB/s wr, 501 op/s 2024-04-02T04:48:16.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:16 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi053"}]: dispatch 2024-04-02T04:48:16.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:16 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-02T04:48:17.320 INFO:teuthology.orchestra.run.smithi053.stdout: 5 159395 11.50 MB/sec execute 207 sec latency 49.685 ms 2024-04-02T04:48:17.415 INFO:teuthology.orchestra.run.smithi053.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-02T04:48:17.415 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager.smithi053 smithi053 *:9093,9094 running (2m) 12s ago 11m 25.7M - 0.23.0 ba2b418f427c bf50c1cd2b3d 2024-04-02T04:48:17.415 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi053 smithi053 running (11m) 12s ago 11m 6409k - 17.2.0 e1d6a67b021e a5c436bbd36d 2024-04-02T04:48:17.415 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi122 smithi122 running (11m) 16s ago 11m 6405k - 17.2.0 e1d6a67b021e b7074474cd46 2024-04-02T04:48:17.415 INFO:teuthology.orchestra.run.smithi053.stdout:grafana.smithi053 smithi053 *:3000 error 12s ago 11m - - 2024-04-02T04:48:17.415 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi053.nfrfot smithi053 running (6m) 12s ago 6m 37.0M - 17.2.0 e1d6a67b021e c94e9e643514 2024-04-02T04:48:17.416 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi122.mwoyoh smithi122 running (5m) 16s ago 5m 15.3M - 17.2.0 e1d6a67b021e 73376d9ee4b8 2024-04-02T04:48:17.416 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi053.legdmv smithi053 *:8443,9283,8765 running (111s) 12s ago 13m 478M - 19.0.0-1979-g6cec793d 78f830190839 c12d516a31bc 2024-04-02T04:48:17.416 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi122.cpwlyk smithi122 *:8443,9283,8765 running (52s) 16s ago 11m 447M - 19.0.0-1979-g6cec793d 78f830190839 9a545c5e7d81 2024-04-02T04:48:17.416 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi053 smithi053 running (32s) 12s ago 13m 33.6M 2048M 19.0.0-1979-g6cec793d 78f830190839 f53a44f19c00 2024-04-02T04:48:17.416 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi122 smithi122 starting - - - 2048M 2024-04-02T04:48:17.416 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.0.0.smithi053.mhkdcb smithi053 *:2049 running (5m) 12s ago 5m 265M - 4.0 e1d6a67b021e 07e5da7bd2c5 2024-04-02T04:48:17.416 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.1.0.smithi122.mauvsb smithi122 *:2049 running (5m) 16s ago 5m 81.2M - 4.0 e1d6a67b021e 4f866685005e 2024-04-02T04:48:17.416 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi053 smithi053 *:9100 running (2m) 12s ago 11m 18.7M - 1.3.1 1dbe0e931976 773b71971e67 2024-04-02T04:48:17.416 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi122 smithi122 *:9100 running (2m) 16s ago 10m 16.5M - 1.3.1 1dbe0e931976 475d439c7980 2024-04-02T04:48:17.416 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 smithi053 running (10m) 12s ago 10m 537M 1937M 17.2.0 e1d6a67b021e 906cb1fd8042 2024-04-02T04:48:17.416 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 smithi053 running (9m) 12s ago 9m 461M 1937M 17.2.0 e1d6a67b021e ac1ad6266e63 2024-04-02T04:48:17.416 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 smithi053 running (9m) 12s ago 9m 367M 1937M 17.2.0 e1d6a67b021e 86964c8a94ff 2024-04-02T04:48:17.416 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 smithi053 running (9m) 12s ago 8m 487M 1937M 17.2.0 e1d6a67b021e 7e8b9e98ae2b 2024-04-02T04:48:17.416 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 smithi122 running (8m) 16s ago 8m 500M 2704M 17.2.0 e1d6a67b021e 46076afcd7bc 2024-04-02T04:48:17.416 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 smithi122 running (8m) 16s ago 8m 562M 2704M 17.2.0 e1d6a67b021e e687a2f6da4d 2024-04-02T04:48:17.417 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 smithi122 running (7m) 16s ago 7m 484M 2704M 17.2.0 e1d6a67b021e 9e371df9a839 2024-04-02T04:48:17.417 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 smithi122 running (7m) 16s ago 7m 353M 2704M 17.2.0 e1d6a67b021e 9b3cc398254d 2024-04-02T04:48:17.417 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus.smithi053 smithi053 *:9095 running (58s) 12s ago 10m 58.7M - 2.33.4 514e6a882f6e 1dd55399da10 2024-04-02T04:48:17.587 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:17 smithi053 ceph-mon[67180]: pgmap v13: 97 pgs: 97 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.7 MiB/s wr, 569 op/s 2024-04-02T04:48:17.587 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:17 smithi053 ceph-mon[67180]: from='client.34130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:48:17.587 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:17 smithi053 ceph-mon[67180]: fsmap foofs:1 {0=foofs.smithi053.nfrfot=up:active} 1 up:standby 2024-04-02T04:48:17.587 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:17 smithi053 ceph-mon[67180]: osdmap e72: 8 total, 8 up, 8 in 2024-04-02T04:48:17.587 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:17 smithi053 ceph-mon[67180]: mgrmap e40: smithi053.legdmv(active, since 19s), standbys: smithi122.cpwlyk 2024-04-02T04:48:17.587 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:17 smithi053 ceph-mon[67180]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-02T04:48:17.587 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:17 smithi053 ceph-mon[67180]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-02T04:48:17.587 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:17 smithi053 ceph-mon[67180]: daemon grafana.smithi053 on smithi053 is in error state 2024-04-02T04:48:17.588 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:17 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi053"}]: dispatch 2024-04-02T04:48:17.588 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:17 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-02T04:48:17.588 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:17 smithi053 ceph-mon[67180]: mon.smithi053 calling monitor election 2024-04-02T04:48:17.588 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:17 smithi053 ceph-mon[67180]: mon.smithi122 calling monitor election 2024-04-02T04:48:17.588 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:17 smithi053 ceph-mon[67180]: mon.smithi053 is new leader, mons smithi053,smithi122 in quorum (ranks 0,1) 2024-04-02T04:48:17.588 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:17 smithi053 ceph-mon[67180]: monmap epoch 3 2024-04-02T04:48:17.588 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:17 smithi053 ceph-mon[67180]: fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 2024-04-02T04:48:17.588 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:17 smithi053 ceph-mon[67180]: last_changed 2024-04-02T04:48:16.391111+0000 2024-04-02T04:48:17.588 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:17 smithi053 ceph-mon[67180]: created 2024-04-02T04:34:55.992242+0000 2024-04-02T04:48:17.588 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:17 smithi053 ceph-mon[67180]: min_mon_release 19 (squid) 2024-04-02T04:48:17.588 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:17 smithi053 ceph-mon[67180]: election_strategy: 1 2024-04-02T04:48:17.588 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:17 smithi053 ceph-mon[67180]: 0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.smithi053 2024-04-02T04:48:17.588 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:17 smithi053 ceph-mon[67180]: 1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.smithi122 2024-04-02T04:48:17.588 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:17 smithi053 ceph-mon[67180]: fsmap foofs:1 {0=foofs.smithi053.nfrfot=up:active} 1 up:standby 2024-04-02T04:48:17.588 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:17 smithi053 ceph-mon[67180]: osdmap e72: 8 total, 8 up, 8 in 2024-04-02T04:48:17.588 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:17 smithi053 ceph-mon[67180]: mgrmap e40: smithi053.legdmv(active, since 19s), standbys: smithi122.cpwlyk 2024-04-02T04:48:17.589 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:17 smithi053 ceph-mon[67180]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-02T04:48:17.589 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:17 smithi053 ceph-mon[67180]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-02T04:48:17.589 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:17 smithi053 ceph-mon[67180]: daemon grafana.smithi053 on smithi053 is in error state 2024-04-02T04:48:17.589 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:17 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:17.589 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:17 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:17.589 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:17 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:48:17.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:17 smithi122 ceph-mon[53211]: pgmap v13: 97 pgs: 97 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.7 MiB/s wr, 569 op/s 2024-04-02T04:48:17.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:17 smithi122 ceph-mon[53211]: from='client.34130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:48:17.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:17 smithi122 ceph-mon[53211]: fsmap foofs:1 {0=foofs.smithi053.nfrfot=up:active} 1 up:standby 2024-04-02T04:48:17.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:17 smithi122 ceph-mon[53211]: osdmap e72: 8 total, 8 up, 8 in 2024-04-02T04:48:17.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:17 smithi122 ceph-mon[53211]: mgrmap e40: smithi053.legdmv(active, since 19s), standbys: smithi122.cpwlyk 2024-04-02T04:48:17.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:17 smithi122 ceph-mon[53211]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-02T04:48:17.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:17 smithi122 ceph-mon[53211]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-02T04:48:17.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:17 smithi122 ceph-mon[53211]: daemon grafana.smithi053 on smithi053 is in error state 2024-04-02T04:48:17.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:17 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi053"}]: dispatch 2024-04-02T04:48:17.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:17 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-02T04:48:17.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:17 smithi122 ceph-mon[53211]: mon.smithi053 calling monitor election 2024-04-02T04:48:17.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:17 smithi122 ceph-mon[53211]: mon.smithi122 calling monitor election 2024-04-02T04:48:17.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:17 smithi122 ceph-mon[53211]: mon.smithi053 is new leader, mons smithi053,smithi122 in quorum (ranks 0,1) 2024-04-02T04:48:17.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:17 smithi122 ceph-mon[53211]: monmap epoch 3 2024-04-02T04:48:17.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:17 smithi122 ceph-mon[53211]: fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 2024-04-02T04:48:17.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:17 smithi122 ceph-mon[53211]: last_changed 2024-04-02T04:48:16.391111+0000 2024-04-02T04:48:17.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:17 smithi122 ceph-mon[53211]: created 2024-04-02T04:34:55.992242+0000 2024-04-02T04:48:17.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:17 smithi122 ceph-mon[53211]: min_mon_release 19 (squid) 2024-04-02T04:48:17.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:17 smithi122 ceph-mon[53211]: election_strategy: 1 2024-04-02T04:48:17.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:17 smithi122 ceph-mon[53211]: 0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.smithi053 2024-04-02T04:48:17.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:17 smithi122 ceph-mon[53211]: 1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.smithi122 2024-04-02T04:48:17.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:17 smithi122 ceph-mon[53211]: fsmap foofs:1 {0=foofs.smithi053.nfrfot=up:active} 1 up:standby 2024-04-02T04:48:17.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:17 smithi122 ceph-mon[53211]: osdmap e72: 8 total, 8 up, 8 in 2024-04-02T04:48:17.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:17 smithi122 ceph-mon[53211]: mgrmap e40: smithi053.legdmv(active, since 19s), standbys: smithi122.cpwlyk 2024-04-02T04:48:17.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:17 smithi122 ceph-mon[53211]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-02T04:48:17.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:17 smithi122 ceph-mon[53211]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-02T04:48:17.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:17 smithi122 ceph-mon[53211]: daemon grafana.smithi053 on smithi053 is in error state 2024-04-02T04:48:17.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:17 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:17.897 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:17 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:17.897 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:17 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:48:18.068 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:48:18.068 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": { 2024-04-02T04:48:18.068 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:48:18.068 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:48:18.068 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": { 2024-04-02T04:48:18.068 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:48:18.068 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:48:18.068 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": { 2024-04-02T04:48:18.069 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-02T04:48:18.069 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:48:18.069 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": { 2024-04-02T04:48:18.069 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:48:18.069 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:48:18.069 INFO:teuthology.orchestra.run.smithi053.stdout: "overall": { 2024-04-02T04:48:18.069 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-04-02T04:48:18.069 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 4 2024-04-02T04:48:18.069 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-02T04:48:18.069 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:48:18.320 INFO:teuthology.orchestra.run.smithi053.stdout: 5 159774 11.49 MB/sec execute 208 sec latency 45.995 ms 2024-04-02T04:48:18.593 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:48:18.594 INFO:teuthology.orchestra.run.smithi053.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae", 2024-04-02T04:48:18.594 INFO:teuthology.orchestra.run.smithi053.stdout: "in_progress": true, 2024-04-02T04:48:18.594 INFO:teuthology.orchestra.run.smithi053.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-02T04:48:18.594 INFO:teuthology.orchestra.run.smithi053.stdout: "services_complete": [ 2024-04-02T04:48:18.594 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr" 2024-04-02T04:48:18.594 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-04-02T04:48:18.594 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": "3/23 daemons upgraded", 2024-04-02T04:48:18.594 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "Currently upgrading mon daemons", 2024-04-02T04:48:18.594 INFO:teuthology.orchestra.run.smithi053.stdout: "is_paused": false 2024-04-02T04:48:18.594 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:48:18.599 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:18 smithi053 ceph-mon[67180]: from='client.34134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:48:18.599 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:18 smithi053 ceph-mon[67180]: from='client.34138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:48:18.599 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:18 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/2443551103' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:48:18.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:18 smithi122 ceph-mon[53211]: from='client.34134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:48:18.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:18 smithi122 ceph-mon[53211]: from='client.34138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:48:18.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:18 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/2443551103' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:48:19.204 INFO:teuthology.orchestra.run.smithi053.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-02T04:48:19.204 INFO:teuthology.orchestra.run.smithi053.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-02T04:48:19.204 INFO:teuthology.orchestra.run.smithi053.stdout: daemon grafana.smithi053 on smithi053 is in error state 2024-04-02T04:48:19.320 INFO:teuthology.orchestra.run.smithi053.stdout: 5 160196 11.47 MB/sec execute 209 sec latency 29.890 ms 2024-04-02T04:48:19.615 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:19 smithi053 ceph-mon[67180]: pgmap v14: 97 pgs: 97 active+clean; 338 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.8 MiB/s wr, 875 op/s 2024-04-02T04:48:19.615 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:19 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/4013067399' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:48:19.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:19 smithi122 ceph-mon[53211]: pgmap v14: 97 pgs: 97 active+clean; 338 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.8 MiB/s wr, 875 op/s 2024-04-02T04:48:19.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:19 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/4013067399' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:48:20.321 INFO:teuthology.orchestra.run.smithi053.stdout: 5 160586 11.47 MB/sec execute 210 sec latency 60.529 ms 2024-04-02T04:48:20.619 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:20 smithi053 ceph-mon[67180]: from='client.34146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:48:20.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:20 smithi122 ceph-mon[53211]: from='client.34146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:48:21.321 INFO:teuthology.orchestra.run.smithi053.stdout: 5 160952 11.46 MB/sec execute 211 sec latency 62.029 ms 2024-04-02T04:48:21.628 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:21 smithi053 ceph-mon[67180]: pgmap v15: 97 pgs: 97 active+clean; 338 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 6.4 MiB/s wr, 594 op/s 2024-04-02T04:48:21.629 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:21 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:21.629 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:21 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:21.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:21 smithi122 ceph-mon[53211]: pgmap v15: 97 pgs: 97 active+clean; 338 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 6.4 MiB/s wr, 594 op/s 2024-04-02T04:48:21.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:21 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:21.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:21 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:22.320 INFO:teuthology.orchestra.run.smithi053.stdout: 5 161365 11.44 MB/sec execute 212 sec latency 43.851 ms 2024-04-02T04:48:22.901 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:22 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:22.901 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:22 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:22.901 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:22 smithi053 ceph-mon[67180]: pgmap v16: 97 pgs: 97 active+clean; 338 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 6.4 MiB/s wr, 594 op/s 2024-04-02T04:48:23.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:22 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:23.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:22 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:23.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:22 smithi122 ceph-mon[53211]: pgmap v16: 97 pgs: 97 active+clean; 338 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 6.4 MiB/s wr, 594 op/s 2024-04-02T04:48:23.320 INFO:teuthology.orchestra.run.smithi053.stdout: 5 161757 11.42 MB/sec execute 213 sec latency 53.023 ms 2024-04-02T04:48:24.321 INFO:teuthology.orchestra.run.smithi053.stdout: 5 162074 11.42 MB/sec execute 214 sec latency 53.682 ms 2024-04-02T04:48:25.321 INFO:teuthology.orchestra.run.smithi053.stdout: 5 162431 11.44 MB/sec execute 215 sec latency 40.280 ms 2024-04-02T04:48:25.446 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:25 smithi053 ceph-mon[67180]: pgmap v17: 97 pgs: 97 active+clean; 331 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.6 MiB/s wr, 842 op/s 2024-04-02T04:48:25.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:25 smithi122 ceph-mon[53211]: pgmap v17: 97 pgs: 97 active+clean; 331 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.6 MiB/s wr, 842 op/s 2024-04-02T04:48:26.321 INFO:teuthology.orchestra.run.smithi053.stdout: 5 162779 11.44 MB/sec execute 216 sec latency 63.655 ms 2024-04-02T04:48:26.713 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:26 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:26.713 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:26 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:26.713 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:26 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:26.713 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:26 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:48:26.713 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:26 smithi053 ceph-mon[67180]: pgmap v18: 97 pgs: 97 active+clean; 331 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.6 MiB/s wr, 587 op/s 2024-04-02T04:48:26.713 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:26 smithi053 ceph-mon[67180]: pgmap v19: 97 pgs: 97 active+clean; 331 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.8 MiB/s wr, 715 op/s 2024-04-02T04:48:26.713 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:26 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:26.713 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:26 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:26.713 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:26 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T04:48:26.713 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:26 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-02T04:48:26.714 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:26 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:26.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:26 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:26.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:26 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:26.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:26 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:26.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:26 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:48:26.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:26 smithi122 ceph-mon[53211]: pgmap v18: 97 pgs: 97 active+clean; 331 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.6 MiB/s wr, 587 op/s 2024-04-02T04:48:26.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:26 smithi122 ceph-mon[53211]: pgmap v19: 97 pgs: 97 active+clean; 331 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.8 MiB/s wr, 715 op/s 2024-04-02T04:48:26.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:26 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:26.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:26 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:26.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:26 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T04:48:26.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:26 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-02T04:48:26.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:26 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:27.321 INFO:teuthology.orchestra.run.smithi053.stdout: 5 163253 11.44 MB/sec execute 217 sec latency 59.682 ms 2024-04-02T04:48:27.717 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:27 smithi053 ceph-mon[67180]: Reconfiguring mon.smithi053 (monmap changed)... 2024-04-02T04:48:27.718 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:27 smithi053 ceph-mon[67180]: Reconfiguring daemon mon.smithi053 on smithi053 2024-04-02T04:48:27.718 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:27 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:48:27.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:27 smithi122 ceph-mon[53211]: Reconfiguring mon.smithi053 (monmap changed)... 2024-04-02T04:48:27.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:27 smithi122 ceph-mon[53211]: Reconfiguring daemon mon.smithi053 on smithi053 2024-04-02T04:48:27.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:27 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:48:28.321 INFO:teuthology.orchestra.run.smithi053.stdout: 5 163824 11.45 MB/sec execute 218 sec latency 31.022 ms 2024-04-02T04:48:28.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:28 smithi053 ceph-mon[67180]: pgmap v20: 97 pgs: 97 active+clean; 331 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 2.9 MiB/s wr, 323 op/s 2024-04-02T04:48:28.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:28 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:28.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:28 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:28.724 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:28 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi053.legdmv", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-02T04:48:28.725 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:28 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-02T04:48:28.725 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:28 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:28.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:28 smithi122 ceph-mon[53211]: pgmap v20: 97 pgs: 97 active+clean; 331 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 2.9 MiB/s wr, 323 op/s 2024-04-02T04:48:28.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:28 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:28.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:28 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:28.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:28 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi053.legdmv", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-02T04:48:28.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:28 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-02T04:48:28.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:28 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:29.321 INFO:teuthology.orchestra.run.smithi053.stdout: 5 164473 11.48 MB/sec execute 219 sec latency 28.327 ms 2024-04-02T04:48:29.728 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:29 smithi053 ceph-mon[67180]: Reconfiguring mgr.smithi053.legdmv (monmap changed)... 2024-04-02T04:48:29.728 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:29 smithi053 ceph-mon[67180]: Reconfiguring daemon mgr.smithi053.legdmv on smithi053 2024-04-02T04:48:29.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:29 smithi122 ceph-mon[53211]: Reconfiguring mgr.smithi053.legdmv (monmap changed)... 2024-04-02T04:48:29.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:29 smithi122 ceph-mon[53211]: Reconfiguring daemon mgr.smithi053.legdmv on smithi053 2024-04-02T04:48:30.321 INFO:teuthology.orchestra.run.smithi053.stdout: 5 165216 11.48 MB/sec execute 220 sec latency 31.367 ms 2024-04-02T04:48:30.733 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:30 smithi053 ceph-mon[67180]: pgmap v21: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 6.4 MiB/s wr, 625 op/s 2024-04-02T04:48:30.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:30 smithi122 ceph-mon[53211]: pgmap v21: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 6.4 MiB/s wr, 625 op/s 2024-04-02T04:48:31.321 INFO:teuthology.orchestra.run.smithi053.stdout: 5 165826 11.52 MB/sec execute 221 sec latency 37.849 ms 2024-04-02T04:48:31.751 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:31 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:31.752 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:31 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:31.752 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:31 smithi053 ceph-mon[67180]: Reconfiguring crash.smithi053 (monmap changed)... 2024-04-02T04:48:31.752 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:31 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi053", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-02T04:48:31.752 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:31 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:31.752 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:31 smithi053 ceph-mon[67180]: Reconfiguring daemon crash.smithi053 on smithi053 2024-04-02T04:48:31.752 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:31 smithi053 ceph-mon[67180]: pgmap v22: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 6.4 MiB/s wr, 625 op/s 2024-04-02T04:48:32.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:31 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:32.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:31 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:32.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:31 smithi122 ceph-mon[53211]: Reconfiguring crash.smithi053 (monmap changed)... 2024-04-02T04:48:32.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:31 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi053", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-02T04:48:32.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:31 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:32.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:31 smithi122 ceph-mon[53211]: Reconfiguring daemon crash.smithi053 on smithi053 2024-04-02T04:48:32.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:31 smithi122 ceph-mon[53211]: pgmap v22: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 6.4 MiB/s wr, 625 op/s 2024-04-02T04:48:32.321 INFO:teuthology.orchestra.run.smithi053.stdout: 5 166486 11.58 MB/sec execute 222 sec latency 30.060 ms 2024-04-02T04:48:33.321 INFO:teuthology.orchestra.run.smithi053.stdout: 5 166792 11.57 MB/sec execute 223 sec latency 38.773 ms 2024-04-02T04:48:33.782 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:33 smithi053 ceph-mon[67180]: pgmap v23: 97 pgs: 97 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 8.4 MiB/s wr, 787 op/s 2024-04-02T04:48:33.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:33 smithi122 ceph-mon[53211]: pgmap v23: 97 pgs: 97 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 8.4 MiB/s wr, 787 op/s 2024-04-02T04:48:34.321 INFO:teuthology.orchestra.run.smithi053.stdout: 5 167108 11.54 MB/sec execute 224 sec latency 53.055 ms 2024-04-02T04:48:35.074 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:34 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:35.074 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:34 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:35.074 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:34 smithi053 ceph-mon[67180]: Reconfiguring osd.0 (monmap changed)... 2024-04-02T04:48:35.074 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:34 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-02T04:48:35.074 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:34 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:35.074 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:34 smithi053 ceph-mon[67180]: Reconfiguring daemon osd.0 on smithi053 2024-04-02T04:48:35.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:34 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:35.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:34 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:35.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:34 smithi122 ceph-mon[53211]: Reconfiguring osd.0 (monmap changed)... 2024-04-02T04:48:35.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:34 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-02T04:48:35.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:34 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:35.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:34 smithi122 ceph-mon[53211]: Reconfiguring daemon osd.0 on smithi053 2024-04-02T04:48:35.321 INFO:teuthology.orchestra.run.smithi053.stdout: 5 167370 11.52 MB/sec execute 225 sec latency 57.716 ms 2024-04-02T04:48:36.085 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:35 smithi053 ceph-mon[67180]: pgmap v24: 97 pgs: 97 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 7.8 MiB/s wr, 724 op/s 2024-04-02T04:48:36.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:35 smithi122 ceph-mon[53211]: pgmap v24: 97 pgs: 97 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 7.8 MiB/s wr, 724 op/s 2024-04-02T04:48:36.321 INFO:teuthology.orchestra.run.smithi053.stdout: 5 167669 11.51 MB/sec execute 226 sec latency 47.474 ms 2024-04-02T04:48:37.321 INFO:teuthology.orchestra.run.smithi053.stdout: 5 167932 11.49 MB/sec execute 227 sec latency 93.706 ms 2024-04-02T04:48:37.325 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:37 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:37.325 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:37 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:37.325 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:37 smithi053 ceph-mon[67180]: Reconfiguring osd.1 (monmap changed)... 2024-04-02T04:48:37.325 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:37 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-02T04:48:37.325 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:37 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:37.325 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:37 smithi053 ceph-mon[67180]: Reconfiguring daemon osd.1 on smithi053 2024-04-02T04:48:37.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:37 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:37.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:37 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:37.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:37 smithi122 ceph-mon[53211]: Reconfiguring osd.1 (monmap changed)... 2024-04-02T04:48:37.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:37 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-02T04:48:37.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:37 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:37.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:37 smithi122 ceph-mon[53211]: Reconfiguring daemon osd.1 on smithi053 2024-04-02T04:48:38.322 INFO:teuthology.orchestra.run.smithi053.stdout: 5 168286 11.45 MB/sec execute 228 sec latency 63.476 ms 2024-04-02T04:48:38.328 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:38 smithi053 ceph-mon[67180]: pgmap v25: 97 pgs: 97 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.5 MiB/s wr, 603 op/s 2024-04-02T04:48:38.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:38 smithi122 ceph-mon[53211]: pgmap v25: 97 pgs: 97 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.5 MiB/s wr, 603 op/s 2024-04-02T04:48:39.322 INFO:teuthology.orchestra.run.smithi053.stdout: 5 168622 11.44 MB/sec execute 229 sec latency 59.747 ms 2024-04-02T04:48:40.127 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:40 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:40.127 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:40 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:40.127 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:40 smithi053 ceph-mon[67180]: Reconfiguring osd.2 (monmap changed)... 2024-04-02T04:48:40.127 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:40 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-02T04:48:40.127 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:40 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:40.127 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:40 smithi053 ceph-mon[67180]: Reconfiguring daemon osd.2 on smithi053 2024-04-02T04:48:40.127 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:40 smithi053 ceph-mon[67180]: pgmap v26: 97 pgs: 97 active+clean; 330 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 799 op/s 2024-04-02T04:48:40.322 INFO:teuthology.orchestra.run.smithi053.stdout: 5 168953 11.42 MB/sec execute 230 sec latency 40.231 ms 2024-04-02T04:48:40.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:40 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:40.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:40 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:40.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:40 smithi122 ceph-mon[53211]: Reconfiguring osd.2 (monmap changed)... 2024-04-02T04:48:40.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:40 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-02T04:48:40.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:40 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:40.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:40 smithi122 ceph-mon[53211]: Reconfiguring daemon osd.2 on smithi053 2024-04-02T04:48:40.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:40 smithi122 ceph-mon[53211]: pgmap v26: 97 pgs: 97 active+clean; 330 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 799 op/s 2024-04-02T04:48:41.322 INFO:teuthology.orchestra.run.smithi053.stdout: 5 169259 11.43 MB/sec execute 231 sec latency 41.929 ms 2024-04-02T04:48:42.322 INFO:teuthology.orchestra.run.smithi053.stdout: 5 169624 11.44 MB/sec execute 232 sec latency 41.642 ms 2024-04-02T04:48:42.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:42 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:42.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:42 smithi053 ceph-mon[67180]: pgmap v27: 97 pgs: 97 active+clean; 330 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.6 MiB/s wr, 567 op/s 2024-04-02T04:48:42.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:42 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:42.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:42 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-02T04:48:42.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:42 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:42.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:42 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:48:42.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:42 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:42.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:42 smithi122 ceph-mon[53211]: pgmap v27: 97 pgs: 97 active+clean; 330 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.6 MiB/s wr, 567 op/s 2024-04-02T04:48:42.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:42 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:42.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:42 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-02T04:48:42.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:42 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:42.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:42 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:48:43.322 INFO:teuthology.orchestra.run.smithi053.stdout: 5 169957 11.44 MB/sec execute 233 sec latency 56.309 ms 2024-04-02T04:48:43.501 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:43 smithi053 ceph-mon[67180]: Reconfiguring osd.3 (monmap changed)... 2024-04-02T04:48:43.501 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:43 smithi053 ceph-mon[67180]: Reconfiguring daemon osd.3 on smithi053 2024-04-02T04:48:43.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:43 smithi122 ceph-mon[53211]: Reconfiguring osd.3 (monmap changed)... 2024-04-02T04:48:43.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:43 smithi122 ceph-mon[53211]: Reconfiguring daemon osd.3 on smithi053 2024-04-02T04:48:44.322 INFO:teuthology.orchestra.run.smithi053.stdout: 5 170199 11.42 MB/sec execute 234 sec latency 48.900 ms 2024-04-02T04:48:44.518 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:44 smithi053 ceph-mon[67180]: pgmap v28: 97 pgs: 97 active+clean; 337 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 7.7 MiB/s wr, 761 op/s 2024-04-02T04:48:44.519 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:44 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:44.519 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:44 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:44.519 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:44 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi053.nfrfot", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T04:48:44.519 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:44 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:44.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:44 smithi122 ceph-mon[53211]: pgmap v28: 97 pgs: 97 active+clean; 337 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 7.7 MiB/s wr, 761 op/s 2024-04-02T04:48:44.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:44 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:44.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:44 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:44.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:44 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi053.nfrfot", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T04:48:44.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:44 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:45.322 INFO:teuthology.orchestra.run.smithi053.stdout: 5 170379 11.39 MB/sec execute 235 sec latency 48.441 ms 2024-04-02T04:48:45.538 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:45 smithi053 ceph-mon[67180]: Reconfiguring mds.foofs.smithi053.nfrfot (monmap changed)... 2024-04-02T04:48:45.538 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:45 smithi053 ceph-mon[67180]: Reconfiguring daemon mds.foofs.smithi053.nfrfot on smithi053 2024-04-02T04:48:45.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:45 smithi122 ceph-mon[53211]: Reconfiguring mds.foofs.smithi053.nfrfot (monmap changed)... 2024-04-02T04:48:45.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:45 smithi122 ceph-mon[53211]: Reconfiguring daemon mds.foofs.smithi053.nfrfot on smithi053 2024-04-02T04:48:46.322 INFO:teuthology.orchestra.run.smithi053.stdout: 5 170652 11.36 MB/sec execute 236 sec latency 51.742 ms 2024-04-02T04:48:46.582 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:46 smithi053 ceph-mon[67180]: pgmap v29: 97 pgs: 97 active+clean; 337 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 3.9 MiB/s wr, 388 op/s 2024-04-02T04:48:46.582 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:46 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:46.582 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:46 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:46.582 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:46 smithi053 ceph-mon[67180]: Reconfiguring crash.smithi122 (monmap changed)... 2024-04-02T04:48:46.582 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:46 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi122", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-02T04:48:46.582 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:46 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:46.582 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:46 smithi053 ceph-mon[67180]: Reconfiguring daemon crash.smithi122 on smithi122 2024-04-02T04:48:46.796 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:46 smithi122 ceph-mon[53211]: pgmap v29: 97 pgs: 97 active+clean; 337 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 3.9 MiB/s wr, 388 op/s 2024-04-02T04:48:46.796 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:46 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:46.796 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:46 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:46.796 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:46 smithi122 ceph-mon[53211]: Reconfiguring crash.smithi122 (monmap changed)... 2024-04-02T04:48:46.796 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:46 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi122", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-02T04:48:46.796 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:46 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:46.796 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:46 smithi122 ceph-mon[53211]: Reconfiguring daemon crash.smithi122 on smithi122 2024-04-02T04:48:47.322 INFO:teuthology.orchestra.run.smithi053.stdout: 5 170861 11.33 MB/sec execute 237 sec latency 77.558 ms 2024-04-02T04:48:47.597 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:47 smithi053 ceph-mon[67180]: pgmap v30: 97 pgs: 97 active+clean; 337 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 3.9 MiB/s wr, 388 op/s 2024-04-02T04:48:47.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:47 smithi122 ceph-mon[53211]: pgmap v30: 97 pgs: 97 active+clean; 337 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 3.9 MiB/s wr, 388 op/s 2024-04-02T04:48:48.322 INFO:teuthology.orchestra.run.smithi053.stdout: 5 171076 11.31 MB/sec execute 238 sec latency 59.141 ms 2024-04-02T04:48:49.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:48 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:49.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:48 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:49.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:48 smithi122 ceph-mon[53211]: Reconfiguring mgr.smithi122.cpwlyk (monmap changed)... 2024-04-02T04:48:49.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:48 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi122.cpwlyk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-02T04:48:49.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:48 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-02T04:48:49.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:48 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:49.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:48 smithi122 ceph-mon[53211]: Reconfiguring daemon mgr.smithi122.cpwlyk on smithi122 2024-04-02T04:48:49.151 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:48 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:49.152 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:48 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:49.152 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:48 smithi053 ceph-mon[67180]: Reconfiguring mgr.smithi122.cpwlyk (monmap changed)... 2024-04-02T04:48:49.152 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:48 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi122.cpwlyk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-02T04:48:49.152 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:48 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-02T04:48:49.152 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:48 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:49.152 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:48 smithi053 ceph-mon[67180]: Reconfiguring daemon mgr.smithi122.cpwlyk on smithi122 2024-04-02T04:48:49.322 INFO:teuthology.orchestra.run.smithi053.stdout: 5 171308 11.29 MB/sec execute 239 sec latency 47.216 ms 2024-04-02T04:48:49.672 INFO:teuthology.orchestra.run.smithi053.stdout:true 2024-04-02T04:48:50.241 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:49 smithi053 ceph-mon[67180]: pgmap v31: 97 pgs: 97 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.6 MiB/s wr, 546 op/s 2024-04-02T04:48:50.241 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:49 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:50.241 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:49 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:50.241 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:49 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T04:48:50.241 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:49 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-02T04:48:50.241 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:49 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:50.322 INFO:teuthology.orchestra.run.smithi053.stdout: 5 171572 11.27 MB/sec execute 240 sec latency 80.655 ms 2024-04-02T04:48:50.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:49 smithi122 ceph-mon[53211]: pgmap v31: 97 pgs: 97 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.6 MiB/s wr, 546 op/s 2024-04-02T04:48:50.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:49 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:50.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:49 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:50.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:49 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T04:48:50.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:49 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-02T04:48:50.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:49 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:50.533 INFO:teuthology.orchestra.run.smithi053.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-02T04:48:50.533 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager.smithi053 smithi053 *:9093,9094 running (3m) 45s ago 12m 25.7M - 0.23.0 ba2b418f427c bf50c1cd2b3d 2024-04-02T04:48:50.533 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi053 smithi053 running (12m) 45s ago 12m 6409k - 17.2.0 e1d6a67b021e a5c436bbd36d 2024-04-02T04:48:50.533 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi122 smithi122 running (11m) 29s ago 11m 6405k - 17.2.0 e1d6a67b021e b7074474cd46 2024-04-02T04:48:50.533 INFO:teuthology.orchestra.run.smithi053.stdout:grafana.smithi053 smithi053 *:3000 error 45s ago 12m - - 2024-04-02T04:48:50.533 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi053.nfrfot smithi053 running (6m) 45s ago 6m 37.0M - 17.2.0 e1d6a67b021e c94e9e643514 2024-04-02T04:48:50.534 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi122.mwoyoh smithi122 running (6m) 29s ago 6m 15.4M - 17.2.0 e1d6a67b021e 73376d9ee4b8 2024-04-02T04:48:50.534 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi053.legdmv smithi053 *:8443,9283,8765 running (2m) 45s ago 13m 478M - 19.0.0-1979-g6cec793d 78f830190839 c12d516a31bc 2024-04-02T04:48:50.534 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi122.cpwlyk smithi122 *:8443,9283,8765 running (85s) 29s ago 11m 448M - 19.0.0-1979-g6cec793d 78f830190839 9a545c5e7d81 2024-04-02T04:48:50.534 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi053 smithi053 running (65s) 45s ago 13m 33.6M 2048M 19.0.0-1979-g6cec793d 78f830190839 f53a44f19c00 2024-04-02T04:48:50.534 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi122 smithi122 running (36s) 29s ago 11m 24.2M 2048M 19.0.0-1979-g6cec793d 78f830190839 44b200fed3c5 2024-04-02T04:48:50.534 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.0.0.smithi053.mhkdcb smithi053 *:2049 running (6m) 45s ago 6m 265M - 4.0 e1d6a67b021e 07e5da7bd2c5 2024-04-02T04:48:50.534 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.1.0.smithi122.mauvsb smithi122 *:2049 running (6m) 29s ago 6m 81.6M - 4.0 e1d6a67b021e 4f866685005e 2024-04-02T04:48:50.534 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi053 smithi053 *:9100 running (2m) 45s ago 12m 18.7M - 1.3.1 1dbe0e931976 773b71971e67 2024-04-02T04:48:50.534 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi122 smithi122 *:9100 running (2m) 29s ago 11m 15.1M - 1.3.1 1dbe0e931976 475d439c7980 2024-04-02T04:48:50.534 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 smithi053 running (10m) 45s ago 10m 537M 1937M 17.2.0 e1d6a67b021e 906cb1fd8042 2024-04-02T04:48:50.535 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 smithi053 running (10m) 45s ago 10m 461M 1937M 17.2.0 e1d6a67b021e ac1ad6266e63 2024-04-02T04:48:50.535 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 smithi053 running (9m) 45s ago 9m 367M 1937M 17.2.0 e1d6a67b021e 86964c8a94ff 2024-04-02T04:48:50.535 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 smithi053 running (9m) 45s ago 9m 487M 1937M 17.2.0 e1d6a67b021e 7e8b9e98ae2b 2024-04-02T04:48:50.535 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 smithi122 running (9m) 29s ago 9m 527M 2704M 17.2.0 e1d6a67b021e 46076afcd7bc 2024-04-02T04:48:50.535 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 smithi122 running (8m) 29s ago 8m 314M 2704M 17.2.0 e1d6a67b021e e687a2f6da4d 2024-04-02T04:48:50.535 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 smithi122 running (8m) 29s ago 8m 508M 2704M 17.2.0 e1d6a67b021e 9e371df9a839 2024-04-02T04:48:50.535 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 smithi122 running (7m) 29s ago 7m 393M 2704M 17.2.0 e1d6a67b021e 9b3cc398254d 2024-04-02T04:48:50.535 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus.smithi053 smithi053 *:9095 running (91s) 45s ago 11m 58.7M - 2.33.4 514e6a882f6e 1dd55399da10 2024-04-02T04:48:51.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:50 smithi122 ceph-mon[53211]: from='client.34154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:48:51.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:50 smithi122 ceph-mon[53211]: Reconfiguring mon.smithi122 (monmap changed)... 2024-04-02T04:48:51.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:50 smithi122 ceph-mon[53211]: Reconfiguring daemon mon.smithi122 on smithi122 2024-04-02T04:48:51.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:50 smithi122 ceph-mon[53211]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:48:51.252 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:50 smithi053 ceph-mon[67180]: from='client.34154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:48:51.253 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:50 smithi053 ceph-mon[67180]: Reconfiguring mon.smithi122 (monmap changed)... 2024-04-02T04:48:51.253 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:50 smithi053 ceph-mon[67180]: Reconfiguring daemon mon.smithi122 on smithi122 2024-04-02T04:48:51.253 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:50 smithi053 ceph-mon[67180]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:48:51.269 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:48:51.269 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": { 2024-04-02T04:48:51.270 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:48:51.270 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:48:51.270 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": { 2024-04-02T04:48:51.270 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:48:51.270 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:48:51.270 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": { 2024-04-02T04:48:51.270 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-02T04:48:51.270 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:48:51.270 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": { 2024-04-02T04:48:51.270 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:48:51.270 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:48:51.271 INFO:teuthology.orchestra.run.smithi053.stdout: "overall": { 2024-04-02T04:48:51.271 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-04-02T04:48:51.271 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 4 2024-04-02T04:48:51.271 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-02T04:48:51.271 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:48:51.322 INFO:teuthology.orchestra.run.smithi053.stdout: 5 171971 11.25 MB/sec execute 241 sec latency 130.060 ms 2024-04-02T04:48:51.878 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:48:51.879 INFO:teuthology.orchestra.run.smithi053.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae", 2024-04-02T04:48:51.879 INFO:teuthology.orchestra.run.smithi053.stdout: "in_progress": true, 2024-04-02T04:48:51.879 INFO:teuthology.orchestra.run.smithi053.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-02T04:48:51.879 INFO:teuthology.orchestra.run.smithi053.stdout: "services_complete": [ 2024-04-02T04:48:51.879 INFO:teuthology.orchestra.run.smithi053.stdout: "mon", 2024-04-02T04:48:51.879 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr" 2024-04-02T04:48:51.879 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-04-02T04:48:51.879 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": "4/23 daemons upgraded", 2024-04-02T04:48:51.879 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "Currently upgrading mon daemons", 2024-04-02T04:48:51.879 INFO:teuthology.orchestra.run.smithi053.stdout: "is_paused": false 2024-04-02T04:48:51.879 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:48:52.024 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:52 smithi053 ceph-mon[67180]: from='client.34162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:48:52.025 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:52 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/253215226' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:48:52.025 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:52 smithi053 ceph-mon[67180]: pgmap v32: 97 pgs: 97 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.7 MiB/s wr, 350 op/s 2024-04-02T04:48:52.025 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:52 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:52.025 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:52 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:52.025 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:52 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-02T04:48:52.025 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:52 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:52.287 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:52 smithi122 ceph-mon[53211]: from='client.34162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:48:52.287 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:52 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/253215226' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:48:52.287 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:52 smithi122 ceph-mon[53211]: pgmap v32: 97 pgs: 97 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.7 MiB/s wr, 350 op/s 2024-04-02T04:48:52.288 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:52 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:52.288 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:52 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:52.288 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:52 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-02T04:48:52.288 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:52 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:52.322 INFO:teuthology.orchestra.run.smithi053.stdout: 5 172323 11.22 MB/sec execute 242 sec latency 42.746 ms 2024-04-02T04:48:52.573 INFO:teuthology.orchestra.run.smithi053.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-02T04:48:52.573 INFO:teuthology.orchestra.run.smithi053.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-02T04:48:52.573 INFO:teuthology.orchestra.run.smithi053.stdout: daemon grafana.smithi053 on smithi053 is in error state 2024-04-02T04:48:53.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:53 smithi053 ceph-mon[67180]: Reconfiguring osd.4 (monmap changed)... 2024-04-02T04:48:53.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:53 smithi053 ceph-mon[67180]: Reconfiguring daemon osd.4 on smithi122 2024-04-02T04:48:53.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:53 smithi053 ceph-mon[67180]: from='client.44109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:48:53.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:53 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/344088327' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:48:53.322 INFO:teuthology.orchestra.run.smithi053.stdout: 5 172740 11.25 MB/sec execute 243 sec latency 37.517 ms 2024-04-02T04:48:53.337 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:53 smithi122 ceph-mon[53211]: Reconfiguring osd.4 (monmap changed)... 2024-04-02T04:48:53.337 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:53 smithi122 ceph-mon[53211]: Reconfiguring daemon osd.4 on smithi122 2024-04-02T04:48:53.337 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:53 smithi122 ceph-mon[53211]: from='client.44109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:48:53.337 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:53 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/344088327' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:48:54.322 INFO:teuthology.orchestra.run.smithi053.stdout: 5 173185 11.27 MB/sec execute 244 sec latency 38.699 ms 2024-04-02T04:48:54.566 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:54.566 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:54.566 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:54 smithi053 ceph-mon[67180]: Reconfiguring osd.5 (monmap changed)... 2024-04-02T04:48:54.566 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-02T04:48:54.566 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:54.566 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:54 smithi053 ceph-mon[67180]: Reconfiguring daemon osd.5 on smithi122 2024-04-02T04:48:54.566 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:54 smithi053 ceph-mon[67180]: pgmap v33: 97 pgs: 97 active+clean; 335 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.1 MiB/s wr, 535 op/s 2024-04-02T04:48:54.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:54.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:54.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:54 smithi122 ceph-mon[53211]: Reconfiguring osd.5 (monmap changed)... 2024-04-02T04:48:54.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-02T04:48:54.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:54.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:54 smithi122 ceph-mon[53211]: Reconfiguring daemon osd.5 on smithi122 2024-04-02T04:48:54.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:54 smithi122 ceph-mon[53211]: pgmap v33: 97 pgs: 97 active+clean; 335 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.1 MiB/s wr, 535 op/s 2024-04-02T04:48:55.322 INFO:teuthology.orchestra.run.smithi053.stdout: 5 173559 11.27 MB/sec execute 245 sec latency 51.932 ms 2024-04-02T04:48:56.322 INFO:teuthology.orchestra.run.smithi053.stdout: 5 173808 11.26 MB/sec execute 246 sec latency 64.654 ms 2024-04-02T04:48:56.376 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:56 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:56.377 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:56 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:56.377 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:56 smithi053 ceph-mon[67180]: Reconfiguring osd.6 (monmap changed)... 2024-04-02T04:48:56.377 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:56 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-02T04:48:56.377 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:56 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:56.377 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:56 smithi053 ceph-mon[67180]: Reconfiguring daemon osd.6 on smithi122 2024-04-02T04:48:56.377 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:56 smithi053 ceph-mon[67180]: pgmap v34: 97 pgs: 97 active+clean; 335 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.1 MiB/s wr, 342 op/s 2024-04-02T04:48:56.624 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:56 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:56.624 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:56 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:56.624 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:56 smithi122 ceph-mon[53211]: Reconfiguring osd.6 (monmap changed)... 2024-04-02T04:48:56.624 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:56 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-02T04:48:56.624 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:56 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:56.624 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:56 smithi122 ceph-mon[53211]: Reconfiguring daemon osd.6 on smithi122 2024-04-02T04:48:56.624 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:56 smithi122 ceph-mon[53211]: pgmap v34: 97 pgs: 97 active+clean; 335 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.1 MiB/s wr, 342 op/s 2024-04-02T04:48:57.322 INFO:teuthology.orchestra.run.smithi053.stdout: 5 174045 11.24 MB/sec execute 247 sec latency 50.586 ms 2024-04-02T04:48:57.388 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:57 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:48:57.388 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:57 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:57.388 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:57 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:57.388 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:57 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-02T04:48:57.388 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:57 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:57.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:57 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:48:57.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:57 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:57.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:57 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:57.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:57 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-02T04:48:57.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:57 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:58.322 INFO:teuthology.orchestra.run.smithi053.stdout: 5 174346 11.21 MB/sec execute 248 sec latency 48.386 ms 2024-04-02T04:48:58.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:58 smithi122 ceph-mon[53211]: Reconfiguring osd.7 (monmap changed)... 2024-04-02T04:48:58.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:58 smithi122 ceph-mon[53211]: Reconfiguring daemon osd.7 on smithi122 2024-04-02T04:48:58.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:58 smithi122 ceph-mon[53211]: pgmap v35: 97 pgs: 97 active+clean; 335 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.1 MiB/s wr, 342 op/s 2024-04-02T04:48:58.401 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:58 smithi053 ceph-mon[67180]: Reconfiguring osd.7 (monmap changed)... 2024-04-02T04:48:58.401 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:58 smithi053 ceph-mon[67180]: Reconfiguring daemon osd.7 on smithi122 2024-04-02T04:48:58.401 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:58 smithi053 ceph-mon[67180]: pgmap v35: 97 pgs: 97 active+clean; 335 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.1 MiB/s wr, 342 op/s 2024-04-02T04:48:59.323 INFO:teuthology.orchestra.run.smithi053.stdout: 5 174621 11.19 MB/sec execute 249 sec latency 62.318 ms 2024-04-02T04:48:59.917 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:59 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:59.917 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:59 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:48:59.917 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:59 smithi053 ceph-mon[67180]: Reconfiguring mds.foofs.smithi122.mwoyoh (monmap changed)... 2024-04-02T04:48:59.917 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:59 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi122.mwoyoh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T04:48:59.917 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:59 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:48:59.917 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:59 smithi053 ceph-mon[67180]: Reconfiguring daemon mds.foofs.smithi122.mwoyoh on smithi122 2024-04-02T04:48:59.917 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:48:59 smithi053 ceph-mon[67180]: pgmap v36: 97 pgs: 97 active+clean; 345 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.5 MiB/s wr, 544 op/s 2024-04-02T04:49:00.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:59 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:00.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:59 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:00.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:59 smithi122 ceph-mon[53211]: Reconfiguring mds.foofs.smithi122.mwoyoh (monmap changed)... 2024-04-02T04:49:00.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:59 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi122.mwoyoh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T04:49:00.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:59 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:49:00.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:59 smithi122 ceph-mon[53211]: Reconfiguring daemon mds.foofs.smithi122.mwoyoh on smithi122 2024-04-02T04:49:00.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:48:59 smithi122 ceph-mon[53211]: pgmap v36: 97 pgs: 97 active+clean; 345 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.5 MiB/s wr, 544 op/s 2024-04-02T04:49:00.323 INFO:teuthology.orchestra.run.smithi053.stdout: 5 174897 11.17 MB/sec execute 250 sec latency 43.430 ms 2024-04-02T04:49:01.323 INFO:teuthology.orchestra.run.smithi053.stdout: 5 175438 11.18 MB/sec execute 251 sec latency 44.898 ms 2024-04-02T04:49:01.943 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:01.943 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:01.943 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:49:01.943 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:49:01.943 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:49:01.943 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:01 smithi053 ceph-mon[67180]: Upgrade: Setting container_image for all mon 2024-04-02T04:49:01.943 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:01.943 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi053"}]: dispatch 2024-04-02T04:49:01.943 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi053"}]': finished 2024-04-02T04:49:01.943 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi122"}]: dispatch 2024-04-02T04:49:01.944 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi122"}]': finished 2024-04-02T04:49:01.944 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:01 smithi053 ceph-mon[67180]: pgmap v37: 97 pgs: 97 active+clean; 345 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.9 MiB/s wr, 386 op/s 2024-04-02T04:49:02.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:02.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:02.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:49:02.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:49:02.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:49:02.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:01 smithi122 ceph-mon[53211]: Upgrade: Setting container_image for all mon 2024-04-02T04:49:02.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:02.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi053"}]: dispatch 2024-04-02T04:49:02.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi053"}]': finished 2024-04-02T04:49:02.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi122"}]: dispatch 2024-04-02T04:49:02.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi122"}]': finished 2024-04-02T04:49:02.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:01 smithi122 ceph-mon[53211]: pgmap v37: 97 pgs: 97 active+clean; 345 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.9 MiB/s wr, 386 op/s 2024-04-02T04:49:02.323 INFO:teuthology.orchestra.run.smithi053.stdout: 5 175907 11.16 MB/sec execute 252 sec latency 23.625 ms 2024-04-02T04:49:03.323 INFO:teuthology.orchestra.run.smithi053.stdout: 5 176372 11.19 MB/sec execute 253 sec latency 50.086 ms 2024-04-02T04:49:03.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:03 smithi053 ceph-mon[67180]: pgmap v38: 97 pgs: 97 active+clean; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.9 MiB/s wr, 625 op/s 2024-04-02T04:49:03.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:03 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:03.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:03 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi053", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-02T04:49:03.706 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:03 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:49:03.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:03 smithi122 ceph-mon[53211]: pgmap v38: 97 pgs: 97 active+clean; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.9 MiB/s wr, 625 op/s 2024-04-02T04:49:03.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:03 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:03.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:03 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi053", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-02T04:49:03.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:03 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:49:04.323 INFO:teuthology.orchestra.run.smithi053.stdout: 5 176789 11.21 MB/sec execute 254 sec latency 36.660 ms 2024-04-02T04:49:04.726 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:04 smithi053 ceph-mon[67180]: Upgrade: Updating crash.smithi053 (1/2) 2024-04-02T04:49:04.727 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:04 smithi053 ceph-mon[67180]: Deploying daemon crash.smithi053 on smithi053 2024-04-02T04:49:04.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:04 smithi122 ceph-mon[53211]: Upgrade: Updating crash.smithi053 (1/2) 2024-04-02T04:49:04.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:04 smithi122 ceph-mon[53211]: Deploying daemon crash.smithi053 on smithi053 2024-04-02T04:49:05.323 INFO:teuthology.orchestra.run.smithi053.stdout: 5 177186 11.21 MB/sec execute 255 sec latency 44.077 ms 2024-04-02T04:49:05.748 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:05 smithi053 ceph-mon[67180]: pgmap v39: 97 pgs: 97 active+clean; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.4 MiB/s wr, 441 op/s 2024-04-02T04:49:05.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:05 smithi122 ceph-mon[53211]: pgmap v39: 97 pgs: 97 active+clean; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.4 MiB/s wr, 441 op/s 2024-04-02T04:49:06.323 INFO:teuthology.orchestra.run.smithi053.stdout: 5 177399 11.19 MB/sec execute 256 sec latency 40.950 ms 2024-04-02T04:49:07.324 INFO:teuthology.orchestra.run.smithi053.stdout: 5 177734 11.18 MB/sec execute 257 sec latency 39.014 ms 2024-04-02T04:49:07.774 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:07 smithi053 ceph-mon[67180]: pgmap v40: 97 pgs: 97 active+clean; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.4 MiB/s wr, 441 op/s 2024-04-02T04:49:07.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:07 smithi122 ceph-mon[53211]: pgmap v40: 97 pgs: 97 active+clean; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.4 MiB/s wr, 441 op/s 2024-04-02T04:49:08.323 INFO:teuthology.orchestra.run.smithi053.stdout: 5 178115 11.17 MB/sec execute 258 sec latency 33.970 ms 2024-04-02T04:49:09.323 INFO:teuthology.orchestra.run.smithi053.stdout: 5 178440 11.16 MB/sec execute 259 sec latency 38.668 ms 2024-04-02T04:49:09.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:09 smithi053 ceph-mon[67180]: pgmap v41: 97 pgs: 97 active+clean; 348 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.3 KiB/s rd, 7.0 MiB/s wr, 655 op/s 2024-04-02T04:49:09.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:09 smithi122 ceph-mon[53211]: pgmap v41: 97 pgs: 97 active+clean; 348 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.3 KiB/s rd, 7.0 MiB/s wr, 655 op/s 2024-04-02T04:49:10.323 INFO:teuthology.orchestra.run.smithi053.stdout: 5 178938 11.16 MB/sec execute 260 sec latency 56.689 ms 2024-04-02T04:49:10.665 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:10 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:10.665 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:10 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:10.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:10 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:10.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:10 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:11.323 INFO:teuthology.orchestra.run.smithi053.stdout: 5 179385 11.14 MB/sec execute 261 sec latency 39.320 ms 2024-04-02T04:49:12.323 INFO:teuthology.orchestra.run.smithi053.stdout: 5 179839 11.16 MB/sec execute 262 sec latency 32.505 ms 2024-04-02T04:49:12.430 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:12 smithi053 ceph-mon[67180]: Upgrade: Updating crash.smithi122 (2/2) 2024-04-02T04:49:12.431 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:12 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:12.431 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:12 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi122", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-02T04:49:12.431 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:12 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:49:12.431 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:12 smithi053 ceph-mon[67180]: Deploying daemon crash.smithi122 on smithi122 2024-04-02T04:49:12.431 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:12 smithi053 ceph-mon[67180]: pgmap v42: 97 pgs: 97 active+clean; 348 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.6 MiB/s wr, 453 op/s 2024-04-02T04:49:12.431 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:12 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:49:12.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:12 smithi122 ceph-mon[53211]: Upgrade: Updating crash.smithi122 (2/2) 2024-04-02T04:49:12.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:12 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:12.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:12 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi122", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-02T04:49:12.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:12 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:49:12.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:12 smithi122 ceph-mon[53211]: Deploying daemon crash.smithi122 on smithi122 2024-04-02T04:49:12.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:12 smithi122 ceph-mon[53211]: pgmap v42: 97 pgs: 97 active+clean; 348 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.6 MiB/s wr, 453 op/s 2024-04-02T04:49:12.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:12 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:49:13.324 INFO:teuthology.orchestra.run.smithi053.stdout: 5 180307 11.19 MB/sec execute 263 sec latency 39.558 ms 2024-04-02T04:49:13.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:13 smithi122 ceph-mon[53211]: pgmap v43: 97 pgs: 97 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.1 MiB/s wr, 717 op/s 2024-04-02T04:49:13.690 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:13 smithi053 ceph-mon[67180]: pgmap v43: 97 pgs: 97 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.1 MiB/s wr, 717 op/s 2024-04-02T04:49:14.324 INFO:teuthology.orchestra.run.smithi053.stdout: 5 180754 11.20 MB/sec execute 264 sec latency 52.630 ms 2024-04-02T04:49:15.324 INFO:teuthology.orchestra.run.smithi053.stdout: 5 181003 11.18 MB/sec execute 265 sec latency 43.065 ms 2024-04-02T04:49:16.324 INFO:teuthology.orchestra.run.smithi053.stdout: 5 181304 11.16 MB/sec execute 266 sec latency 50.414 ms 2024-04-02T04:49:16.481 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:16 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:16.481 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:16 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:16.481 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:16 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:49:16.481 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:16 smithi053 ceph-mon[67180]: pgmap v44: 97 pgs: 97 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.1 MiB/s wr, 478 op/s 2024-04-02T04:49:16.546 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:16 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:16.546 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:16 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:16.546 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:16 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:49:16.546 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:16 smithi122 ceph-mon[53211]: pgmap v44: 97 pgs: 97 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.1 MiB/s wr, 478 op/s 2024-04-02T04:49:17.324 INFO:teuthology.orchestra.run.smithi053.stdout: 5 181638 11.14 MB/sec execute 267 sec latency 41.674 ms 2024-04-02T04:49:17.744 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:17 smithi053 ceph-mon[67180]: pgmap v45: 97 pgs: 97 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.1 MiB/s wr, 478 op/s 2024-04-02T04:49:17.838 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:17 smithi122 ceph-mon[53211]: pgmap v45: 97 pgs: 97 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.1 MiB/s wr, 478 op/s 2024-04-02T04:49:18.324 INFO:teuthology.orchestra.run.smithi053.stdout: 5 182150 11.16 MB/sec execute 268 sec latency 43.343 ms 2024-04-02T04:49:19.324 INFO:teuthology.orchestra.run.smithi053.stdout: 5 182748 11.15 MB/sec execute 269 sec latency 39.349 ms 2024-04-02T04:49:20.324 INFO:teuthology.orchestra.run.smithi053.stdout: 5 183130 11.14 MB/sec execute 270 sec latency 47.311 ms 2024-04-02T04:49:20.516 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:20 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:20.516 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:20 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:20.516 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:20 smithi053 ceph-mon[67180]: pgmap v46: 97 pgs: 97 active+clean; 345 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 7.5 MiB/s wr, 703 op/s 2024-04-02T04:49:20.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:20 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:20.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:20 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:20.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:20 smithi122 ceph-mon[53211]: pgmap v46: 97 pgs: 97 active+clean; 345 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 7.5 MiB/s wr, 703 op/s 2024-04-02T04:49:21.324 INFO:teuthology.orchestra.run.smithi053.stdout: 5 183427 11.15 MB/sec execute 271 sec latency 36.949 ms 2024-04-02T04:49:21.798 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:21 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:21.798 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:21 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:21.798 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:21 smithi053 ceph-mon[67180]: pgmap v47: 97 pgs: 97 active+clean; 345 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.0 MiB/s wr, 489 op/s 2024-04-02T04:49:21.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:21 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:21.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:21 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:21.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:21 smithi122 ceph-mon[53211]: pgmap v47: 97 pgs: 97 active+clean; 345 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.0 MiB/s wr, 489 op/s 2024-04-02T04:49:22.324 INFO:teuthology.orchestra.run.smithi053.stdout: 5 183766 11.15 MB/sec execute 272 sec latency 60.799 ms 2024-04-02T04:49:23.125 INFO:teuthology.orchestra.run.smithi053.stdout:true 2024-04-02T04:49:23.324 INFO:teuthology.orchestra.run.smithi053.stdout: 5 184067 11.15 MB/sec execute 273 sec latency 51.810 ms 2024-04-02T04:49:23.821 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:23 smithi053 ceph-mon[67180]: from='client.34178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:49:23.822 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:23 smithi053 ceph-mon[67180]: pgmap v48: 97 pgs: 97 active+clean; 344 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.3 MiB/s wr, 736 op/s 2024-04-02T04:49:23.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:23 smithi122 ceph-mon[53211]: from='client.34178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:49:23.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:23 smithi122 ceph-mon[53211]: pgmap v48: 97 pgs: 97 active+clean; 344 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.3 MiB/s wr, 736 op/s 2024-04-02T04:49:24.149 INFO:teuthology.orchestra.run.smithi053.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-02T04:49:24.149 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager.smithi053 smithi053 *:9093,9094 running (4m) 0s ago 12m 25.7M - 0.23.0 ba2b418f427c bf50c1cd2b3d 2024-04-02T04:49:24.149 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi053 smithi053 running (15s) 0s ago 12m 6660k - 19.0.0-1979-g6cec793d 78f830190839 4ce8915e0157 2024-04-02T04:49:24.149 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi122 smithi122 running (9s) 4s ago 12m 6660k - 19.0.0-1979-g6cec793d 78f830190839 e2b5369e6fdf 2024-04-02T04:49:24.149 INFO:teuthology.orchestra.run.smithi053.stdout:grafana.smithi053 smithi053 *:3000 error 0s ago 12m - - 2024-04-02T04:49:24.149 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi053.nfrfot smithi053 running (7m) 0s ago 7m 37.2M - 17.2.0 e1d6a67b021e c94e9e643514 2024-04-02T04:49:24.149 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi122.mwoyoh smithi122 running (7m) 4s ago 7m 15.6M - 17.2.0 e1d6a67b021e 73376d9ee4b8 2024-04-02T04:49:24.149 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi053.legdmv smithi053 *:8443,9283,8765 running (2m) 0s ago 14m 500M - 19.0.0-1979-g6cec793d 78f830190839 c12d516a31bc 2024-04-02T04:49:24.150 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi122.cpwlyk smithi122 *:8443,9283,8765 running (118s) 4s ago 12m 449M - 19.0.0-1979-g6cec793d 78f830190839 9a545c5e7d81 2024-04-02T04:49:24.150 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi053 smithi053 running (99s) 0s ago 14m 44.6M 2048M 19.0.0-1979-g6cec793d 78f830190839 f53a44f19c00 2024-04-02T04:49:24.150 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi122 smithi122 running (69s) 4s ago 12m 34.8M 2048M 19.0.0-1979-g6cec793d 78f830190839 44b200fed3c5 2024-04-02T04:49:24.150 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.0.0.smithi053.mhkdcb smithi053 *:2049 running (6m) 0s ago 6m 271M - 4.0 e1d6a67b021e 07e5da7bd2c5 2024-04-02T04:49:24.150 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.1.0.smithi122.mauvsb smithi122 *:2049 running (6m) 4s ago 6m 81.7M - 4.0 e1d6a67b021e 4f866685005e 2024-04-02T04:49:24.150 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi053 smithi053 *:9100 running (3m) 0s ago 12m 20.4M - 1.3.1 1dbe0e931976 773b71971e67 2024-04-02T04:49:24.150 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi122 smithi122 *:9100 running (3m) 4s ago 12m 15.7M - 1.3.1 1dbe0e931976 475d439c7980 2024-04-02T04:49:24.150 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 smithi053 running (11m) 0s ago 11m 378M 1937M 17.2.0 e1d6a67b021e 906cb1fd8042 2024-04-02T04:49:24.150 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 smithi053 running (10m) 0s ago 10m 557M 1937M 17.2.0 e1d6a67b021e ac1ad6266e63 2024-04-02T04:49:24.150 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 smithi053 running (10m) 0s ago 10m 464M 1937M 17.2.0 e1d6a67b021e 86964c8a94ff 2024-04-02T04:49:24.150 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 smithi053 running (10m) 0s ago 10m 557M 1937M 17.2.0 e1d6a67b021e 7e8b9e98ae2b 2024-04-02T04:49:24.150 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 smithi122 running (9m) 4s ago 9m 557M 2704M 17.2.0 e1d6a67b021e 46076afcd7bc 2024-04-02T04:49:24.150 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 smithi122 running (9m) 4s ago 9m 389M 2704M 17.2.0 e1d6a67b021e e687a2f6da4d 2024-04-02T04:49:24.150 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 smithi122 running (8m) 4s ago 8m 568M 2704M 17.2.0 e1d6a67b021e 9e371df9a839 2024-04-02T04:49:24.150 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 smithi122 running (8m) 4s ago 8m 480M 2704M 17.2.0 e1d6a67b021e 9b3cc398254d 2024-04-02T04:49:24.151 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus.smithi053 smithi053 *:9095 running (2m) 0s ago 11m 67.3M - 2.33.4 514e6a882f6e 1dd55399da10 2024-04-02T04:49:24.324 INFO:teuthology.orchestra.run.smithi053.stdout: 5 184315 11.14 MB/sec execute 274 sec latency 63.180 ms 2024-04-02T04:49:24.841 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:24 smithi053 ceph-mon[67180]: from='client.44119 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:49:24.841 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:24 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:24.841 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:24 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:24.841 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:24 smithi053 ceph-mon[67180]: from='client.34186 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:49:24.909 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:49:24.909 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": { 2024-04-02T04:49:24.909 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:49:24.909 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:49:24.909 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": { 2024-04-02T04:49:24.909 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:49:24.909 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:49:24.909 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": { 2024-04-02T04:49:24.909 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-02T04:49:24.909 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:49:24.909 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": { 2024-04-02T04:49:24.910 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:49:24.910 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:49:24.910 INFO:teuthology.orchestra.run.smithi053.stdout: "overall": { 2024-04-02T04:49:24.910 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-04-02T04:49:24.910 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 4 2024-04-02T04:49:24.910 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-02T04:49:24.910 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:49:25.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:24 smithi122 ceph-mon[53211]: from='client.44119 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:49:25.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:24 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:25.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:24 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:25.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:24 smithi122 ceph-mon[53211]: from='client.34186 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:49:25.324 INFO:teuthology.orchestra.run.smithi053.stdout: 5 184771 11.14 MB/sec execute 275 sec latency 34.816 ms 2024-04-02T04:49:25.482 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:49:25.482 INFO:teuthology.orchestra.run.smithi053.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae", 2024-04-02T04:49:25.482 INFO:teuthology.orchestra.run.smithi053.stdout: "in_progress": true, 2024-04-02T04:49:25.482 INFO:teuthology.orchestra.run.smithi053.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-02T04:49:25.482 INFO:teuthology.orchestra.run.smithi053.stdout: "services_complete": [ 2024-04-02T04:49:25.482 INFO:teuthology.orchestra.run.smithi053.stdout: "mon", 2024-04-02T04:49:25.482 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr", 2024-04-02T04:49:25.483 INFO:teuthology.orchestra.run.smithi053.stdout: "crash" 2024-04-02T04:49:25.483 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-04-02T04:49:25.483 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": "6/23 daemons upgraded", 2024-04-02T04:49:25.483 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "Currently upgrading crash daemons", 2024-04-02T04:49:25.483 INFO:teuthology.orchestra.run.smithi053.stdout: "is_paused": false 2024-04-02T04:49:25.483 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:49:25.857 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:25.857 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:25.857 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:25 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/3341050053' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:49:25.857 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:25 smithi053 ceph-mon[67180]: from='client.34194 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:49:25.857 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:25 smithi053 ceph-mon[67180]: pgmap v49: 97 pgs: 97 active+clean; 344 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.8 MiB/s wr, 473 op/s 2024-04-02T04:49:25.857 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:25.857 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:26.141 INFO:teuthology.orchestra.run.smithi053.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-02T04:49:26.141 INFO:teuthology.orchestra.run.smithi053.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-02T04:49:26.141 INFO:teuthology.orchestra.run.smithi053.stdout: daemon grafana.smithi053 on smithi053 is in error state 2024-04-02T04:49:26.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:26.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:26.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:25 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/3341050053' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:49:26.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:25 smithi122 ceph-mon[53211]: from='client.34194 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:49:26.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:25 smithi122 ceph-mon[53211]: pgmap v49: 97 pgs: 97 active+clean; 344 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.8 MiB/s wr, 473 op/s 2024-04-02T04:49:26.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:26.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:26.324 INFO:teuthology.orchestra.run.smithi053.stdout: 5 185155 11.13 MB/sec execute 276 sec latency 40.310 ms 2024-04-02T04:49:26.872 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:26 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/3364847990' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:49:26.872 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:26 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:49:27.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:26 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/3364847990' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:49:27.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:26 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:49:27.324 INFO:teuthology.orchestra.run.smithi053.stdout: 5 185386 11.11 MB/sec execute 277 sec latency 52.845 ms 2024-04-02T04:49:27.896 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:27 smithi053 ceph-mon[67180]: pgmap v50: 97 pgs: 97 active+clean; 344 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.8 MiB/s wr, 473 op/s 2024-04-02T04:49:28.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:27 smithi122 ceph-mon[53211]: pgmap v50: 97 pgs: 97 active+clean; 344 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.8 MiB/s wr, 473 op/s 2024-04-02T04:49:28.325 INFO:teuthology.orchestra.run.smithi053.stdout: 5 185701 11.11 MB/sec execute 278 sec latency 45.962 ms 2024-04-02T04:49:29.324 INFO:teuthology.orchestra.run.smithi053.stdout: 5 186092 11.08 MB/sec execute 279 sec latency 69.971 ms 2024-04-02T04:49:29.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:29 smithi053 ceph-mon[67180]: mgrmap e41: smithi053.legdmv(active, since 92s), standbys: smithi122.cpwlyk 2024-04-02T04:49:29.934 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:29 smithi053 ceph-mon[67180]: pgmap v51: 97 pgs: 97 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.1 MiB/s wr, 686 op/s 2024-04-02T04:49:30.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:29 smithi122 ceph-mon[53211]: mgrmap e41: smithi053.legdmv(active, since 92s), standbys: smithi122.cpwlyk 2024-04-02T04:49:30.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:29 smithi122 ceph-mon[53211]: pgmap v51: 97 pgs: 97 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.1 MiB/s wr, 686 op/s 2024-04-02T04:49:30.325 INFO:teuthology.orchestra.run.smithi053.stdout: 5 186451 11.06 MB/sec execute 280 sec latency 68.924 ms 2024-04-02T04:49:31.325 INFO:teuthology.orchestra.run.smithi053.stdout: 5 186906 11.08 MB/sec execute 281 sec latency 44.273 ms 2024-04-02T04:49:31.964 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:31 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:31.964 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:31 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:31.964 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:31 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:49:31.964 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:31 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:49:31.964 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:31 smithi053 ceph-mon[67180]: pgmap v52: 97 pgs: 97 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.9 MiB/s wr, 491 op/s 2024-04-02T04:49:31.965 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:31 smithi053 ceph-mon[67180]: pgmap v53: 97 pgs: 97 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.4 KiB/s rd, 6.0 MiB/s wr, 598 op/s 2024-04-02T04:49:31.965 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:31 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:31.965 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:31 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:31.965 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:31 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:49:31.965 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:31 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:49:31.965 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:31 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:49:31.965 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:31 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:49:31.965 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:31 smithi053 ceph-mon[67180]: Upgrade: Setting container_image for all crash 2024-04-02T04:49:31.965 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:31 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:31.965 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:31 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi053"}]: dispatch 2024-04-02T04:49:31.965 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:31 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi053"}]': finished 2024-04-02T04:49:31.965 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:31 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi122"}]: dispatch 2024-04-02T04:49:31.965 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:31 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi122"}]': finished 2024-04-02T04:49:31.965 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:31 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-02T04:49:31.966 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:31 smithi053 ceph-mon[67180]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-02T04:49:31.966 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:31 smithi053 ceph-mon[67180]: Upgrade: osd.0 is safe to restart 2024-04-02T04:49:32.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:31 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:32.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:31 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:32.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:31 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:49:32.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:31 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:49:32.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:31 smithi122 ceph-mon[53211]: pgmap v52: 97 pgs: 97 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.9 MiB/s wr, 491 op/s 2024-04-02T04:49:32.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:31 smithi122 ceph-mon[53211]: pgmap v53: 97 pgs: 97 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.4 KiB/s rd, 6.0 MiB/s wr, 598 op/s 2024-04-02T04:49:32.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:31 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:32.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:31 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:32.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:31 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:49:32.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:31 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:49:32.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:31 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:49:32.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:31 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:49:32.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:31 smithi122 ceph-mon[53211]: Upgrade: Setting container_image for all crash 2024-04-02T04:49:32.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:31 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:32.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:31 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi053"}]: dispatch 2024-04-02T04:49:32.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:31 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi053"}]': finished 2024-04-02T04:49:32.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:31 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi122"}]: dispatch 2024-04-02T04:49:32.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:31 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi122"}]': finished 2024-04-02T04:49:32.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:31 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-02T04:49:32.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:31 smithi122 ceph-mon[53211]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-02T04:49:32.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:31 smithi122 ceph-mon[53211]: Upgrade: osd.0 is safe to restart 2024-04-02T04:49:32.325 INFO:teuthology.orchestra.run.smithi053.stdout: 5 187581 11.13 MB/sec execute 282 sec latency 31.987 ms 2024-04-02T04:49:33.325 INFO:teuthology.orchestra.run.smithi053.stdout: 5 187953 11.14 MB/sec execute 283 sec latency 37.061 ms 2024-04-02T04:49:33.989 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:33 smithi053 ceph-mon[67180]: pgmap v54: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.1 MiB/s wr, 602 op/s 2024-04-02T04:49:34.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:33 smithi122 ceph-mon[53211]: pgmap v54: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.1 MiB/s wr, 602 op/s 2024-04-02T04:49:34.325 INFO:teuthology.orchestra.run.smithi053.stdout: 5 188275 11.12 MB/sec execute 284 sec latency 56.202 ms 2024-04-02T04:49:35.014 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:34 smithi053 ceph-mon[67180]: Upgrade: Updating osd.0 2024-04-02T04:49:35.014 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:34 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:35.014 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:34 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-02T04:49:35.015 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:34 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:49:35.015 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:34 smithi053 ceph-mon[67180]: Deploying daemon osd.0 on smithi053 2024-04-02T04:49:35.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:34 smithi122 ceph-mon[53211]: Upgrade: Updating osd.0 2024-04-02T04:49:35.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:34 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:35.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:34 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-02T04:49:35.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:34 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:49:35.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:34 smithi122 ceph-mon[53211]: Deploying daemon osd.0 on smithi053 2024-04-02T04:49:35.325 INFO:teuthology.orchestra.run.smithi053.stdout: 5 188560 11.10 MB/sec execute 285 sec latency 62.616 ms 2024-04-02T04:49:36.028 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:35 smithi053 ceph-mon[67180]: pgmap v55: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.1 MiB/s wr, 602 op/s 2024-04-02T04:49:36.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:35 smithi122 ceph-mon[53211]: pgmap v55: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.1 MiB/s wr, 602 op/s 2024-04-02T04:49:36.325 INFO:teuthology.orchestra.run.smithi053.stdout: 5 188967 11.10 MB/sec execute 286 sec latency 31.343 ms 2024-04-02T04:49:37.325 INFO:teuthology.orchestra.run.smithi053.stdout: 5 189364 11.09 MB/sec execute 287 sec latency 48.981 ms 2024-04-02T04:49:38.045 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:37 smithi053 ceph-mon[67180]: pgmap v56: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.1 MiB/s wr, 602 op/s 2024-04-02T04:49:38.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:37 smithi122 ceph-mon[53211]: pgmap v56: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.1 MiB/s wr, 602 op/s 2024-04-02T04:49:38.325 INFO:teuthology.orchestra.run.smithi053.stdout: 5 189766 11.08 MB/sec execute 288 sec latency 69.019 ms 2024-04-02T04:49:39.114 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:39 smithi053 systemd[1]: Stopping Ceph osd.0 for 48e16742-f0aa-11ee-b647-cb9ed24678a4... 2024-04-02T04:49:39.325 INFO:teuthology.orchestra.run.smithi053.stdout: 5 190004 11.05 MB/sec execute 289 sec latency 65.755 ms 2024-04-02T04:49:39.641 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:39 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0[35320]: 2024-04-02T04:49:39.401+0000 7f96b6b3e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T04:49:39.641 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:39 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0[35320]: 2024-04-02T04:49:39.401+0000 7f96b6b3e700 -1 osd.0 72 *** Got signal Terminated *** 2024-04-02T04:49:39.641 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:39 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0[35320]: 2024-04-02T04:49:39.401+0000 7f96b6b3e700 -1 osd.0 72 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T04:49:39.893 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:39 smithi053 ceph-mon[67180]: pgmap v57: 97 pgs: 97 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.1 KiB/s rd, 6.1 MiB/s wr, 630 op/s 2024-04-02T04:49:39.893 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:39 smithi053 ceph-mon[67180]: osd.0 marked itself down and dead 2024-04-02T04:49:40.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:39 smithi122 ceph-mon[53211]: pgmap v57: 97 pgs: 97 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.1 KiB/s rd, 6.1 MiB/s wr, 630 op/s 2024-04-02T04:49:40.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:39 smithi122 ceph-mon[53211]: osd.0 marked itself down and dead 2024-04-02T04:49:40.325 INFO:teuthology.orchestra.run.smithi053.stdout: 5 190199 11.02 MB/sec execute 290 sec latency 422.784 ms 2024-04-02T04:49:40.807 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:40 smithi053 podman[72661]: 2024-04-02 04:49:40.558176639 +0000 UTC m=+1.287857493 container died 906cb1fd80422afad23aa12dc1746454736f17aa0fe8ac9b98822ba6649dd0dc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, io.openshift.tags=base centos centos-stream, architecture=x86_64, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, GIT_CLEAN=True, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, name=centos-stream, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux ) 2024-04-02T04:49:41.135 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:40 smithi053 podman[72661]: 2024-04-02 04:49:40.995578012 +0000 UTC m=+1.725258866 container cleanup 906cb1fd80422afad23aa12dc1746454736f17aa0fe8ac9b98822ba6649dd0dc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, distribution-scope=public, io.openshift.tags=base centos centos-stream, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., name=centos-stream, com.redhat.component=centos-stream-container, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-02T04:49:41.135 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:40 smithi053 bash[72661]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0 2024-04-02T04:49:41.135 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:40 smithi053 ceph-mon[67180]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-02T04:49:41.135 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:40 smithi053 ceph-mon[67180]: failed to encode map e73 with expected crc 2024-04-02T04:49:41.135 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:40 smithi053 ceph-mon[67180]: osdmap e73: 8 total, 7 up, 8 in 2024-04-02T04:49:41.135 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:40 smithi053 ceph-mon[67180]: failed to encode map e73 with expected crc 2024-04-02T04:49:41.135 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:40 smithi053 ceph-mon[67180]: failed to encode map e73 with expected crc 2024-04-02T04:49:41.135 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:40 smithi053 ceph-mon[67180]: failed to encode map e73 with expected crc 2024-04-02T04:49:41.135 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:40 smithi053 ceph-mon[67180]: failed to encode map e73 with expected crc 2024-04-02T04:49:41.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:40 smithi122 ceph-mon[53211]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-02T04:49:41.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:40 smithi122 ceph-mon[53211]: failed to encode map e73 with expected crc 2024-04-02T04:49:41.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:40 smithi122 ceph-mon[53211]: osdmap e73: 8 total, 7 up, 8 in 2024-04-02T04:49:41.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:40 smithi122 ceph-mon[53211]: failed to encode map e73 with expected crc 2024-04-02T04:49:41.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:40 smithi122 ceph-mon[53211]: failed to encode map e73 with expected crc 2024-04-02T04:49:41.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:40 smithi122 ceph-mon[53211]: failed to encode map e73 with expected crc 2024-04-02T04:49:41.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:40 smithi122 ceph-mon[53211]: failed to encode map e73 with expected crc 2024-04-02T04:49:41.325 INFO:teuthology.orchestra.run.smithi053.stdout: 5 190392 11.03 MB/sec execute 291 sec latency 1040.575 ms 2024-04-02T04:49:41.392 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:41 smithi053 podman[72680]: 2024-04-02 04:49:41.256125702 +0000 UTC m=+0.702257240 container remove 906cb1fd80422afad23aa12dc1746454736f17aa0fe8ac9b98822ba6649dd0dc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, name=centos-stream, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, release=754, ceph=True, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, io.buildah.version=1.19.8) 2024-04-02T04:49:41.766 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:41 smithi053 podman[72768]: 2024-04-02 04:49:41.757303327 +0000 UTC m=+0.116476429 container create aaed21668ae2f3692c9fe2e5741e36e75f66f00c480fdd20b54527f14eb52220 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-6cec793) 2024-04-02T04:49:41.767 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:41 smithi053 podman[72768]: 2024-04-02 04:49:41.664132405 +0000 UTC m=+0.023305536 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:49:42.022 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:41 smithi053 ceph-mon[67180]: pgmap v59: 97 pgs: 13 stale+active+clean, 84 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.6 MiB/s wr, 582 op/s 2024-04-02T04:49:42.022 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:41 smithi053 ceph-mon[67180]: failed to encode map e74 with expected crc 2024-04-02T04:49:42.022 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:41 smithi053 ceph-mon[67180]: osdmap e74: 8 total, 7 up, 8 in 2024-04-02T04:49:42.022 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:41 smithi053 ceph-mon[67180]: failed to encode map e74 with expected crc 2024-04-02T04:49:42.022 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:41 smithi053 ceph-mon[67180]: failed to encode map e74 with expected crc 2024-04-02T04:49:42.022 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:41 smithi053 ceph-mon[67180]: failed to encode map e74 with expected crc 2024-04-02T04:49:42.022 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:41 smithi053 ceph-mon[67180]: failed to encode map e74 with expected crc 2024-04-02T04:49:42.022 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:41 smithi053 ceph-mon[67180]: failed to encode map e74 with expected crc 2024-04-02T04:49:42.022 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:41 smithi053 ceph-mon[67180]: failed to encode map e74 with expected crc 2024-04-02T04:49:42.022 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:41 smithi053 ceph-mon[67180]: failed to encode map e74 with expected crc 2024-04-02T04:49:42.023 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:41 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:42.023 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:41 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:49:42.284 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:42 smithi053 podman[72768]: 2024-04-02 04:49:42.089048156 +0000 UTC m=+0.448221273 container init aaed21668ae2f3692c9fe2e5741e36e75f66f00c480fdd20b54527f14eb52220 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-6cec793, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-02T04:49:42.284 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:42 smithi053 podman[72768]: 2024-04-02 04:49:42.095435588 +0000 UTC m=+0.454608698 container start aaed21668ae2f3692c9fe2e5741e36e75f66f00c480fdd20b54527f14eb52220 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-6cec793, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-02T04:49:42.284 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:42 smithi053 podman[72768]: 2024-04-02 04:49:42.211486532 +0000 UTC m=+0.570659652 container attach aaed21668ae2f3692c9fe2e5741e36e75f66f00c480fdd20b54527f14eb52220 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-6cec793, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS) 2024-04-02T04:49:42.325 INFO:teuthology.orchestra.run.smithi053.stdout: 5 190597 11.01 MB/sec execute 292 sec latency 61.213 ms 2024-04-02T04:49:42.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:41 smithi122 ceph-mon[53211]: pgmap v59: 97 pgs: 13 stale+active+clean, 84 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.6 MiB/s wr, 582 op/s 2024-04-02T04:49:42.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:41 smithi122 ceph-mon[53211]: failed to encode map e74 with expected crc 2024-04-02T04:49:42.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:41 smithi122 ceph-mon[53211]: osdmap e74: 8 total, 7 up, 8 in 2024-04-02T04:49:42.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:41 smithi122 ceph-mon[53211]: failed to encode map e74 with expected crc 2024-04-02T04:49:42.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:41 smithi122 ceph-mon[53211]: failed to encode map e74 with expected crc 2024-04-02T04:49:42.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:41 smithi122 ceph-mon[53211]: failed to encode map e74 with expected crc 2024-04-02T04:49:42.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:41 smithi122 ceph-mon[53211]: failed to encode map e74 with expected crc 2024-04-02T04:49:42.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:41 smithi122 ceph-mon[53211]: failed to encode map e74 with expected crc 2024-04-02T04:49:42.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:41 smithi122 ceph-mon[53211]: failed to encode map e74 with expected crc 2024-04-02T04:49:42.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:41 smithi122 ceph-mon[53211]: failed to encode map e74 with expected crc 2024-04-02T04:49:42.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:41 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:42.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:41 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:49:42.551 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:42 smithi053 podman[72768]: 2024-04-02 04:49:42.497695602 +0000 UTC m=+0.856868708 container died aaed21668ae2f3692c9fe2e5741e36e75f66f00c480fdd20b54527f14eb52220 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0-deactivate, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-6cec793, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T04:49:43.072 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:42 smithi053 ceph-mon[67180]: Health check failed: Degraded data redundancy: 198/1551 objects degraded (12.766%), 29 pgs degraded (PG_DEGRADED) 2024-04-02T04:49:43.325 INFO:teuthology.orchestra.run.smithi053.stdout: 5 190972 11.02 MB/sec execute 293 sec latency 52.306 ms 2024-04-02T04:49:43.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:42 smithi122 ceph-mon[53211]: Health check failed: Degraded data redundancy: 198/1551 objects degraded (12.766%), 29 pgs degraded (PG_DEGRADED) 2024-04-02T04:49:43.573 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:43 smithi053 podman[72794]: 2024-04-02 04:49:43.564281337 +0000 UTC m=+1.059635910 container remove aaed21668ae2f3692c9fe2e5741e36e75f66f00c480fdd20b54527f14eb52220 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0-deactivate, RELEASE=squid-6cec793, ceph=True, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-02T04:49:43.833 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:43 smithi053 systemd[1]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.0.service: Deactivated successfully. 2024-04-02T04:49:43.833 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:43 smithi053 systemd[1]: Stopped Ceph osd.0 for 48e16742-f0aa-11ee-b647-cb9ed24678a4. 2024-04-02T04:49:43.834 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:43 smithi053 systemd[1]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.0.service: Consumed 3min 17.204s CPU time. 2024-04-02T04:49:44.083 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:43 smithi053 ceph-mon[67180]: pgmap v61: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 343 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.7 MiB/s wr, 576 op/s; 198/1551 objects degraded (12.766%) 2024-04-02T04:49:44.084 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:43 smithi053 systemd[1]: Starting Ceph osd.0 for 48e16742-f0aa-11ee-b647-cb9ed24678a4... 2024-04-02T04:49:44.325 INFO:teuthology.orchestra.run.smithi053.stdout: 5 191319 11.02 MB/sec execute 294 sec latency 48.393 ms 2024-04-02T04:49:44.335 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:44 smithi053 podman[72885]: 2024-04-02 04:49:44.109169445 +0000 UTC m=+0.027899510 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:49:44.335 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:44 smithi053 podman[72885]: 2024-04-02 04:49:44.221344127 +0000 UTC m=+0.140074183 container create a3d170660dd01fce58754b2991a87162d21bfafea339beb71f7a77e37c3a15fa (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0-activate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-6cec793, org.label-schema.build-date=20240326, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-02T04:49:44.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:43 smithi122 ceph-mon[53211]: pgmap v61: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 343 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.7 MiB/s wr, 576 op/s; 198/1551 objects degraded (12.766%) 2024-04-02T04:49:44.586 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:44 smithi053 podman[72885]: 2024-04-02 04:49:44.492010794 +0000 UTC m=+0.410740875 container init a3d170660dd01fce58754b2991a87162d21bfafea339beb71f7a77e37c3a15fa (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0-activate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-6cec793, GIT_CLEAN=True) 2024-04-02T04:49:44.586 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:44 smithi053 podman[72885]: 2024-04-02 04:49:44.500854404 +0000 UTC m=+0.419584467 container start a3d170660dd01fce58754b2991a87162d21bfafea339beb71f7a77e37c3a15fa (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240326, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-6cec793, org.label-schema.schema-version=1.0) 2024-04-02T04:49:44.839 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:44 smithi053 podman[72885]: 2024-04-02 04:49:44.638143464 +0000 UTC m=+0.556873520 container attach a3d170660dd01fce58754b2991a87162d21bfafea339beb71f7a77e37c3a15fa (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0-activate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-6cec793, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T04:49:45.325 INFO:teuthology.orchestra.run.smithi053.stdout: 5 191577 11.01 MB/sec execute 295 sec latency 37.643 ms 2024-04-02T04:49:45.343 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:45 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0-activate[72898]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-02T04:49:45.343 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:45 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0-activate[72898]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-02T04:49:45.343 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:45 smithi053 bash[72885]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-02T04:49:45.343 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:45 smithi053 bash[72885]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-02T04:49:45.343 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:45 smithi053 bash[72885]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-02T04:49:45.343 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:45 smithi053 bash[72885]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-02T04:49:45.343 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:45 smithi053 bash[72885]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-02T04:49:45.343 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:45 smithi053 bash[72885]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-02T04:49:45.344 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:45 smithi053 bash[72885]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-02T04:49:45.344 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:45 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0-activate[72898]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-02T04:49:45.344 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:45 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0-activate[72898]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-02T04:49:45.344 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:45 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0-activate[72898]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-02T04:49:45.344 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:45 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0-activate[72898]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-02T04:49:45.344 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:45 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0-activate[72898]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-02T04:49:45.344 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:45 smithi053 podman[72885]: 2024-04-02 04:49:45.161398138 +0000 UTC m=+1.080128210 container died a3d170660dd01fce58754b2991a87162d21bfafea339beb71f7a77e37c3a15fa (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0-activate, org.label-schema.build-date=20240326, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, RELEASE=squid-6cec793, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T04:49:46.132 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:45 smithi053 ceph-mon[67180]: pgmap v62: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 343 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.7 MiB/s wr, 576 op/s; 198/1551 objects degraded (12.766%) 2024-04-02T04:49:46.132 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:45 smithi053 podman[73025]: 2024-04-02 04:49:45.972925716 +0000 UTC m=+0.805020924 container remove a3d170660dd01fce58754b2991a87162d21bfafea339beb71f7a77e37c3a15fa (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0-activate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-6cec793, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T04:49:46.214 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:45 smithi122 ceph-mon[53211]: pgmap v62: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 343 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.7 MiB/s wr, 576 op/s; 198/1551 objects degraded (12.766%) 2024-04-02T04:49:46.325 INFO:teuthology.orchestra.run.smithi053.stdout: 5 191878 10.99 MB/sec execute 296 sec latency 41.714 ms 2024-04-02T04:49:46.383 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:46 smithi053 podman[73083]: 2024-04-02 04:49:46.146229267 +0000 UTC m=+0.029193011 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:49:46.383 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:46 smithi053 podman[73083]: 2024-04-02 04:49:46.274237041 +0000 UTC m=+0.157200762 container create 490c346c76395bb103c58d16b72777fbb92f3be978fd3c8df6c017825fb4f9a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-6cec793, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T04:49:46.888 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:46 smithi053 podman[73083]: 2024-04-02 04:49:46.755253645 +0000 UTC m=+0.638217377 container init 490c346c76395bb103c58d16b72777fbb92f3be978fd3c8df6c017825fb4f9a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, RELEASE=squid-6cec793, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-02T04:49:46.888 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:46 smithi053 podman[73083]: 2024-04-02 04:49:46.76602494 +0000 UTC m=+0.648988660 container start 490c346c76395bb103c58d16b72777fbb92f3be978fd3c8df6c017825fb4f9a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-6cec793, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.build-date=20240326) 2024-04-02T04:49:46.888 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:46 smithi053 ceph-osd[73100]: -- 172.21.15.53:0/1198272031 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x5614c94e5ba0 con 0x5614c94b7000 2024-04-02T04:49:46.888 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:46 smithi053 ceph-osd[73100]: -- 172.21.15.53:0/1198272031 <== mon.1 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5614ca2a21e0 con 0x5614c94b7000 2024-04-02T04:49:47.137 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:46 smithi053 bash[73083]: 490c346c76395bb103c58d16b72777fbb92f3be978fd3c8df6c017825fb4f9a3 2024-04-02T04:49:47.137 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:46 smithi053 systemd[1]: Started Ceph osd.0 for 48e16742-f0aa-11ee-b647-cb9ed24678a4. 2024-04-02T04:49:47.325 INFO:teuthology.orchestra.run.smithi053.stdout: 5 192264 10.98 MB/sec execute 297 sec latency 49.783 ms 2024-04-02T04:49:47.387 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:47 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0[73096]: 2024-04-02T04:49:47.345+0000 7fc3f81767c0 -1 Falling back to public interface 2024-04-02T04:49:48.155 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:48 smithi053 ceph-mon[67180]: pgmap v63: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 343 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 2.4 KiB/s rd, 2.4 MiB/s wr, 224 op/s; 198/1551 objects degraded (12.766%) 2024-04-02T04:49:48.155 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:48 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:48.155 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:48 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:48.155 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:48 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:49:48.325 INFO:teuthology.orchestra.run.smithi053.stdout: 5 192688 11.00 MB/sec execute 298 sec latency 46.316 ms 2024-04-02T04:49:48.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:48 smithi122 ceph-mon[53211]: pgmap v63: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 343 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 2.4 KiB/s rd, 2.4 MiB/s wr, 224 op/s; 198/1551 objects degraded (12.766%) 2024-04-02T04:49:48.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:48 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:48.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:48 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:48.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:48 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:49:49.171 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:49 smithi053 ceph-mon[67180]: Health check update: Degraded data redundancy: 228/1776 objects degraded (12.838%), 29 pgs degraded (PG_DEGRADED) 2024-04-02T04:49:49.325 INFO:teuthology.orchestra.run.smithi053.stdout: 5 193092 10.97 MB/sec execute 299 sec latency 65.253 ms 2024-04-02T04:49:49.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:49 smithi122 ceph-mon[53211]: Health check update: Degraded data redundancy: 228/1776 objects degraded (12.838%), 29 pgs degraded (PG_DEGRADED) 2024-04-02T04:49:50.194 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:50 smithi053 ceph-mon[67180]: pgmap v64: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 357 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 5.3 MiB/s wr, 478 op/s; 228/1776 objects degraded (12.838%) 2024-04-02T04:49:50.325 INFO:teuthology.orchestra.run.smithi053.stdout: 5 193438 10.96 MB/sec execute 300 sec latency 59.278 ms 2024-04-02T04:49:50.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:50 smithi122 ceph-mon[53211]: pgmap v64: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 357 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 5.3 MiB/s wr, 478 op/s; 228/1776 objects degraded (12.838%) 2024-04-02T04:49:51.325 INFO:teuthology.orchestra.run.smithi053.stdout: 5 193681 10.94 MB/sec execute 301 sec latency 46.507 ms 2024-04-02T04:49:52.221 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:52 smithi053 ceph-mon[67180]: pgmap v65: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 357 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.7 MiB/s wr, 423 op/s; 228/1776 objects degraded (12.838%) 2024-04-02T04:49:52.325 INFO:teuthology.orchestra.run.smithi053.stdout: 5 193982 10.95 MB/sec execute 302 sec latency 55.350 ms 2024-04-02T04:49:52.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:52 smithi122 ceph-mon[53211]: pgmap v65: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 357 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.7 MiB/s wr, 423 op/s; 228/1776 objects degraded (12.838%) 2024-04-02T04:49:52.737 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:52 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0[73096]: 2024-04-02T04:49:52.493+0000 7fc3f81767c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-04-02T04:49:53.325 INFO:teuthology.orchestra.run.smithi053.stdout: 5 194281 10.94 MB/sec execute 303 sec latency 47.231 ms 2024-04-02T04:49:54.022 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:53 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0[73096]: 2024-04-02T04:49:53.954+0000 7fc3f81767c0 -1 osd.0 72 log_to_monitors true 2024-04-02T04:49:54.273 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:49:54 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0[73096]: 2024-04-02T04:49:54.168+0000 7fc3e95ce700 -1 osd.0 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T04:49:54.273 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:54 smithi053 ceph-mon[67180]: pgmap v66: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 346 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.8 MiB/s wr, 571 op/s; 201/1539 objects degraded (13.060%) 2024-04-02T04:49:54.273 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:54 smithi053 ceph-mon[67180]: from='osd.0 [v2:172.21.15.53:6802/1036326940,v1:172.21.15.53:6803/1036326940]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-02T04:49:54.273 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:54 smithi053 ceph-mon[67180]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-02T04:49:54.325 INFO:teuthology.orchestra.run.smithi053.stdout: 5 194677 10.96 MB/sec execute 304 sec latency 42.753 ms 2024-04-02T04:49:54.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:54 smithi122 ceph-mon[53211]: pgmap v66: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 346 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.8 MiB/s wr, 571 op/s; 201/1539 objects degraded (13.060%) 2024-04-02T04:49:54.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:54 smithi122 ceph-mon[53211]: from='osd.0 [v2:172.21.15.53:6802/1036326940,v1:172.21.15.53:6803/1036326940]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-02T04:49:54.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:54 smithi122 ceph-mon[53211]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-02T04:49:55.303 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:55 smithi053 ceph-mon[67180]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-02T04:49:55.303 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:55 smithi053 ceph-mon[67180]: failed to encode map e75 with expected crc 2024-04-02T04:49:55.303 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:55 smithi053 ceph-mon[67180]: osdmap e75: 8 total, 7 up, 8 in 2024-04-02T04:49:55.303 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:55 smithi053 ceph-mon[67180]: failed to encode map e75 with expected crc 2024-04-02T04:49:55.303 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:55 smithi053 ceph-mon[67180]: from='osd.0 [v2:172.21.15.53:6802/1036326940,v1:172.21.15.53:6803/1036326940]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-02T04:49:55.303 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:55 smithi053 ceph-mon[67180]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-02T04:49:55.325 INFO:teuthology.orchestra.run.smithi053.stdout: 5 194940 10.95 MB/sec execute 305 sec latency 141.188 ms 2024-04-02T04:49:55.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:55 smithi122 ceph-mon[53211]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-02T04:49:55.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:55 smithi122 ceph-mon[53211]: failed to encode map e75 with expected crc 2024-04-02T04:49:55.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:55 smithi122 ceph-mon[53211]: osdmap e75: 8 total, 7 up, 8 in 2024-04-02T04:49:55.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:55 smithi122 ceph-mon[53211]: failed to encode map e75 with expected crc 2024-04-02T04:49:55.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:55 smithi122 ceph-mon[53211]: from='osd.0 [v2:172.21.15.53:6802/1036326940,v1:172.21.15.53:6803/1036326940]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-02T04:49:55.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:55 smithi122 ceph-mon[53211]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-02T04:49:56.325 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:56 smithi053 ceph-mon[67180]: pgmap v68: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.9 MiB/s wr, 498 op/s; 201/1539 objects degraded (13.060%) 2024-04-02T04:49:56.325 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:56 smithi053 ceph-mon[67180]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-02T04:49:56.325 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:56 smithi053 ceph-mon[67180]: failed to encode map e76 with expected crc 2024-04-02T04:49:56.326 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:56 smithi053 ceph-mon[67180]: osd.0 [v2:172.21.15.53:6802/1036326940,v1:172.21.15.53:6803/1036326940] boot 2024-04-02T04:49:56.326 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:56 smithi053 ceph-mon[67180]: osdmap e76: 8 total, 8 up, 8 in 2024-04-02T04:49:56.326 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:56 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:49:56.326 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:56 smithi053 ceph-mon[67180]: failed to encode map e76 with expected crc 2024-04-02T04:49:56.326 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:56 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:56.326 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:56 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:56.326 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:56 smithi053 ceph-mon[67180]: Health check update: Degraded data redundancy: 201/1539 objects degraded (13.060%), 29 pgs degraded (PG_DEGRADED) 2024-04-02T04:49:56.326 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:56 smithi053 ceph-mon[67180]: osdmap e77: 8 total, 8 up, 8 in 2024-04-02T04:49:56.326 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:56 smithi053 ceph-mon[67180]: failed to encode map e77 with expected crc 2024-04-02T04:49:56.326 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:56 smithi053 ceph-mon[67180]: failed to encode map e77 with expected crc 2024-04-02T04:49:56.327 INFO:teuthology.orchestra.run.smithi053.stdout: 5 195141 10.93 MB/sec execute 306 sec latency 620.204 ms 2024-04-02T04:49:56.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:56 smithi122 ceph-mon[53211]: pgmap v68: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.9 MiB/s wr, 498 op/s; 201/1539 objects degraded (13.060%) 2024-04-02T04:49:56.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:56 smithi122 ceph-mon[53211]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-02T04:49:56.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:56 smithi122 ceph-mon[53211]: failed to encode map e76 with expected crc 2024-04-02T04:49:56.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:56 smithi122 ceph-mon[53211]: osd.0 [v2:172.21.15.53:6802/1036326940,v1:172.21.15.53:6803/1036326940] boot 2024-04-02T04:49:56.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:56 smithi122 ceph-mon[53211]: osdmap e76: 8 total, 8 up, 8 in 2024-04-02T04:49:56.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:56 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T04:49:56.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:56 smithi122 ceph-mon[53211]: failed to encode map e76 with expected crc 2024-04-02T04:49:56.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:56 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:56.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:56 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:56.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:56 smithi122 ceph-mon[53211]: Health check update: Degraded data redundancy: 201/1539 objects degraded (13.060%), 29 pgs degraded (PG_DEGRADED) 2024-04-02T04:49:56.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:56 smithi122 ceph-mon[53211]: osdmap e77: 8 total, 8 up, 8 in 2024-04-02T04:49:56.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:56 smithi122 ceph-mon[53211]: failed to encode map e77 with expected crc 2024-04-02T04:49:56.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:56 smithi122 ceph-mon[53211]: failed to encode map e77 with expected crc 2024-04-02T04:49:56.722 INFO:teuthology.orchestra.run.smithi053.stdout:true 2024-04-02T04:49:57.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:57 smithi122 ceph-mon[53211]: failed to encode map e77 with expected crc 2024-04-02T04:49:57.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:57 smithi122 ceph-mon[53211]: failed to encode map e77 with expected crc 2024-04-02T04:49:57.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:57 smithi122 ceph-mon[53211]: failed to encode map e77 with expected crc 2024-04-02T04:49:57.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:57 smithi122 ceph-mon[53211]: failed to encode map e77 with expected crc 2024-04-02T04:49:57.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:57 smithi122 ceph-mon[53211]: failed to encode map e77 with expected crc 2024-04-02T04:49:57.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:57 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:57.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:57 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:49:57.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:57 smithi122 ceph-mon[53211]: failed to encode map e78 with expected crc 2024-04-02T04:49:57.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:57 smithi122 ceph-mon[53211]: failed to encode map e78 with expected crc 2024-04-02T04:49:57.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:57 smithi122 ceph-mon[53211]: osdmap e78: 8 total, 8 up, 8 in 2024-04-02T04:49:57.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:57 smithi122 ceph-mon[53211]: failed to encode map e78 with expected crc 2024-04-02T04:49:57.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:57 smithi122 ceph-mon[53211]: failed to encode map e78 with expected crc 2024-04-02T04:49:57.325 INFO:teuthology.orchestra.run.smithi053.stdout: 5 195414 10.90 MB/sec execute 307 sec latency 401.649 ms 2024-04-02T04:49:57.338 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:57 smithi053 ceph-mon[67180]: failed to encode map e77 with expected crc 2024-04-02T04:49:57.338 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:57 smithi053 ceph-mon[67180]: failed to encode map e77 with expected crc 2024-04-02T04:49:57.338 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:57 smithi053 ceph-mon[67180]: failed to encode map e77 with expected crc 2024-04-02T04:49:57.338 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:57 smithi053 ceph-mon[67180]: failed to encode map e77 with expected crc 2024-04-02T04:49:57.338 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:57 smithi053 ceph-mon[67180]: failed to encode map e77 with expected crc 2024-04-02T04:49:57.338 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:57 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:57.338 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:57 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:49:57.338 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:57 smithi053 ceph-mon[67180]: failed to encode map e78 with expected crc 2024-04-02T04:49:57.338 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:57 smithi053 ceph-mon[67180]: failed to encode map e78 with expected crc 2024-04-02T04:49:57.338 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:57 smithi053 ceph-mon[67180]: osdmap e78: 8 total, 8 up, 8 in 2024-04-02T04:49:57.338 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:57 smithi053 ceph-mon[67180]: failed to encode map e78 with expected crc 2024-04-02T04:49:57.338 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:57 smithi053 ceph-mon[67180]: failed to encode map e78 with expected crc 2024-04-02T04:49:57.969 INFO:teuthology.orchestra.run.smithi053.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-02T04:49:57.970 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager.smithi053 smithi053 *:9093,9094 running (4m) 2s ago 13m 24.3M - 0.23.0 ba2b418f427c bf50c1cd2b3d 2024-04-02T04:49:57.970 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi053 smithi053 running (48s) 2s ago 13m 6660k - 19.0.0-1979-g6cec793d 78f830190839 4ce8915e0157 2024-04-02T04:49:57.970 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi122 smithi122 running (43s) 38s ago 12m 6660k - 19.0.0-1979-g6cec793d 78f830190839 e2b5369e6fdf 2024-04-02T04:49:57.970 INFO:teuthology.orchestra.run.smithi053.stdout:grafana.smithi053 smithi053 *:3000 error 2s ago 13m - - 2024-04-02T04:49:57.970 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi053.nfrfot smithi053 running (7m) 2s ago 7m 37.4M - 17.2.0 e1d6a67b021e c94e9e643514 2024-04-02T04:49:57.970 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi122.mwoyoh smithi122 running (7m) 38s ago 7m 15.6M - 17.2.0 e1d6a67b021e 73376d9ee4b8 2024-04-02T04:49:57.970 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi053.legdmv smithi053 *:8443,9283,8765 running (3m) 2s ago 14m 502M - 19.0.0-1979-g6cec793d 78f830190839 c12d516a31bc 2024-04-02T04:49:57.970 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi122.cpwlyk smithi122 *:8443,9283,8765 running (2m) 38s ago 12m 449M - 19.0.0-1979-g6cec793d 78f830190839 9a545c5e7d81 2024-04-02T04:49:57.970 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi053 smithi053 running (2m) 2s ago 14m 47.3M 2048M 19.0.0-1979-g6cec793d 78f830190839 f53a44f19c00 2024-04-02T04:49:57.970 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi122 smithi122 running (103s) 38s ago 12m 34.8M 2048M 19.0.0-1979-g6cec793d 78f830190839 44b200fed3c5 2024-04-02T04:49:57.970 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.0.0.smithi053.mhkdcb smithi053 *:2049 running (7m) 2s ago 7m 274M - 4.0 e1d6a67b021e 07e5da7bd2c5 2024-04-02T04:49:57.970 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.1.0.smithi122.mauvsb smithi122 *:2049 running (7m) 38s ago 7m 81.7M - 4.0 e1d6a67b021e 4f866685005e 2024-04-02T04:49:57.970 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi053 smithi053 *:9100 running (4m) 2s ago 13m 18.7M - 1.3.1 1dbe0e931976 773b71971e67 2024-04-02T04:49:57.970 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi122 smithi122 *:9100 running (3m) 38s ago 12m 15.7M - 1.3.1 1dbe0e931976 475d439c7980 2024-04-02T04:49:57.971 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 smithi053 running (11s) 2s ago 12m 14.0M 1937M 19.0.0-1979-g6cec793d 78f830190839 490c346c7639 2024-04-02T04:49:57.971 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 smithi053 running (11m) 2s ago 11m 587M 1937M 17.2.0 e1d6a67b021e ac1ad6266e63 2024-04-02T04:49:57.971 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 smithi053 running (11m) 2s ago 11m 506M 1937M 17.2.0 e1d6a67b021e 86964c8a94ff 2024-04-02T04:49:57.971 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 smithi053 running (10m) 2s ago 10m 597M 1937M 17.2.0 e1d6a67b021e 7e8b9e98ae2b 2024-04-02T04:49:57.971 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 smithi122 running (10m) 38s ago 10m 557M 2704M 17.2.0 e1d6a67b021e 46076afcd7bc 2024-04-02T04:49:57.971 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 smithi122 running (9m) 38s ago 9m 389M 2704M 17.2.0 e1d6a67b021e e687a2f6da4d 2024-04-02T04:49:57.971 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 smithi122 running (9m) 38s ago 9m 568M 2704M 17.2.0 e1d6a67b021e 9e371df9a839 2024-04-02T04:49:57.971 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 smithi122 running (9m) 38s ago 9m 480M 2704M 17.2.0 e1d6a67b021e 9b3cc398254d 2024-04-02T04:49:57.971 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus.smithi053 smithi053 *:9095 running (2m) 2s ago 12m 67.3M - 2.33.4 514e6a882f6e 1dd55399da10 2024-04-02T04:49:58.325 INFO:teuthology.orchestra.run.smithi053.stdout: 5 195679 10.89 MB/sec execute 308 sec latency 411.489 ms 2024-04-02T04:49:58.349 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:58 smithi053 ceph-mon[67180]: from='client.34206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:49:58.349 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:58 smithi053 ceph-mon[67180]: failed to encode map e78 with expected crc 2024-04-02T04:49:58.349 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:58 smithi053 ceph-mon[67180]: failed to encode map e78 with expected crc 2024-04-02T04:49:58.349 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:58 smithi053 ceph-mon[67180]: failed to encode map e78 with expected crc 2024-04-02T04:49:58.349 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:58 smithi053 ceph-mon[67180]: failed to encode map e78 with expected crc 2024-04-02T04:49:58.349 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:58 smithi053 ceph-mon[67180]: pgmap v72: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 201/1539 objects degraded (13.060%) 2024-04-02T04:49:58.349 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:58 smithi053 ceph-mon[67180]: failed to encode map e78 with expected crc 2024-04-02T04:49:58.349 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:58 smithi053 ceph-mon[67180]: from='client.34210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:49:58.349 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:58 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:58.349 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:58 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:58.349 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:58 smithi053 ceph-mon[67180]: failed to encode map e79 with expected crc 2024-04-02T04:49:58.349 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:58 smithi053 ceph-mon[67180]: osdmap e79: 8 total, 8 up, 8 in 2024-04-02T04:49:58.349 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:58 smithi053 ceph-mon[67180]: failed to encode map e79 with expected crc 2024-04-02T04:49:58.350 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:58 smithi053 ceph-mon[67180]: failed to encode map e79 with expected crc 2024-04-02T04:49:58.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:58 smithi122 ceph-mon[53211]: from='client.34206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:49:58.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:58 smithi122 ceph-mon[53211]: failed to encode map e78 with expected crc 2024-04-02T04:49:58.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:58 smithi122 ceph-mon[53211]: failed to encode map e78 with expected crc 2024-04-02T04:49:58.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:58 smithi122 ceph-mon[53211]: failed to encode map e78 with expected crc 2024-04-02T04:49:58.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:58 smithi122 ceph-mon[53211]: failed to encode map e78 with expected crc 2024-04-02T04:49:58.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:58 smithi122 ceph-mon[53211]: pgmap v72: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 201/1539 objects degraded (13.060%) 2024-04-02T04:49:58.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:58 smithi122 ceph-mon[53211]: failed to encode map e78 with expected crc 2024-04-02T04:49:58.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:58 smithi122 ceph-mon[53211]: from='client.34210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:49:58.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:58 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:58.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:58 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:49:58.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:58 smithi122 ceph-mon[53211]: failed to encode map e79 with expected crc 2024-04-02T04:49:58.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:58 smithi122 ceph-mon[53211]: osdmap e79: 8 total, 8 up, 8 in 2024-04-02T04:49:58.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:58 smithi122 ceph-mon[53211]: failed to encode map e79 with expected crc 2024-04-02T04:49:58.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:58 smithi122 ceph-mon[53211]: failed to encode map e79 with expected crc 2024-04-02T04:49:58.562 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:49:58.562 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": { 2024-04-02T04:49:58.563 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:49:58.563 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:49:58.563 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": { 2024-04-02T04:49:58.563 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:49:58.563 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:49:58.563 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": { 2024-04-02T04:49:58.563 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-04-02T04:49:58.563 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 1 2024-04-02T04:49:58.563 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:49:58.563 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": { 2024-04-02T04:49:58.563 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:49:58.563 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:49:58.563 INFO:teuthology.orchestra.run.smithi053.stdout: "overall": { 2024-04-02T04:49:58.563 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-04-02T04:49:58.564 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 5 2024-04-02T04:49:58.564 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-02T04:49:58.564 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:49:59.138 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:49:59.138 INFO:teuthology.orchestra.run.smithi053.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae", 2024-04-02T04:49:59.138 INFO:teuthology.orchestra.run.smithi053.stdout: "in_progress": true, 2024-04-02T04:49:59.138 INFO:teuthology.orchestra.run.smithi053.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-02T04:49:59.138 INFO:teuthology.orchestra.run.smithi053.stdout: "services_complete": [ 2024-04-02T04:49:59.138 INFO:teuthology.orchestra.run.smithi053.stdout: "mon", 2024-04-02T04:49:59.138 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr", 2024-04-02T04:49:59.138 INFO:teuthology.orchestra.run.smithi053.stdout: "crash" 2024-04-02T04:49:59.138 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-04-02T04:49:59.138 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": "7/23 daemons upgraded", 2024-04-02T04:49:59.138 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "Currently upgrading osd daemons", 2024-04-02T04:49:59.139 INFO:teuthology.orchestra.run.smithi053.stdout: "is_paused": false 2024-04-02T04:49:59.139 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:49:59.325 INFO:teuthology.orchestra.run.smithi053.stdout: 5 195973 10.88 MB/sec execute 309 sec latency 838.037 ms 2024-04-02T04:49:59.377 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:59 smithi053 ceph-mon[67180]: failed to encode map e79 with expected crc 2024-04-02T04:49:59.377 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:59 smithi053 ceph-mon[67180]: failed to encode map e79 with expected crc 2024-04-02T04:49:59.377 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:59 smithi053 ceph-mon[67180]: failed to encode map e79 with expected crc 2024-04-02T04:49:59.377 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:59 smithi053 ceph-mon[67180]: from='client.34214 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:49:59.377 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:59 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/3589922943' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:49:59.377 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:59 smithi053 ceph-mon[67180]: osdmap e80: 8 total, 8 up, 8 in 2024-04-02T04:49:59.377 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:59 smithi053 ceph-mon[67180]: failed to encode map e80 with expected crc 2024-04-02T04:49:59.377 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:49:59 smithi053 ceph-mon[67180]: failed to encode map e80 with expected crc 2024-04-02T04:49:59.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:59 smithi122 ceph-mon[53211]: failed to encode map e79 with expected crc 2024-04-02T04:49:59.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:59 smithi122 ceph-mon[53211]: failed to encode map e79 with expected crc 2024-04-02T04:49:59.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:59 smithi122 ceph-mon[53211]: failed to encode map e79 with expected crc 2024-04-02T04:49:59.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:59 smithi122 ceph-mon[53211]: from='client.34214 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:49:59.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:59 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/3589922943' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:49:59.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:59 smithi122 ceph-mon[53211]: osdmap e80: 8 total, 8 up, 8 in 2024-04-02T04:49:59.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:59 smithi122 ceph-mon[53211]: failed to encode map e80 with expected crc 2024-04-02T04:49:59.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:49:59 smithi122 ceph-mon[53211]: failed to encode map e80 with expected crc 2024-04-02T04:49:59.802 INFO:teuthology.orchestra.run.smithi053.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Degraded data redundancy: 103/1689 objects degraded (6.098%), 9 pgs degraded 2024-04-02T04:49:59.802 INFO:teuthology.orchestra.run.smithi053.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-02T04:49:59.802 INFO:teuthology.orchestra.run.smithi053.stdout: daemon grafana.smithi053 on smithi053 is in error state 2024-04-02T04:49:59.802 INFO:teuthology.orchestra.run.smithi053.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 103/1689 objects degraded (6.098%), 9 pgs degraded 2024-04-02T04:49:59.802 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.13 is active+undersized+degraded, acting [4,3] 2024-04-02T04:49:59.802 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.1f is active+undersized+degraded, acting [7,4] 2024-04-02T04:49:59.802 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.1 is active+undersized+degraded, acting [4,3] 2024-04-02T04:49:59.802 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.7 is active+recovery_wait+degraded, acting [3,7,0] 2024-04-02T04:49:59.802 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.b is active+recovering+degraded, acting [3,0,4] 2024-04-02T04:49:59.802 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.12 is active+recovery_wait+undersized+degraded+remapped, acting [3,7] 2024-04-02T04:49:59.803 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.17 is active+undersized+degraded, acting [5,3] 2024-04-02T04:49:59.803 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.1b is active+undersized+degraded, acting [4,7] 2024-04-02T04:49:59.803 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.1f is active+undersized+degraded, acting [5,2] 2024-04-02T04:50:00.326 INFO:teuthology.orchestra.run.smithi053.stdout: 5 196425 10.88 MB/sec execute 310 sec latency 95.468 ms 2024-04-02T04:50:00.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:00 smithi122 ceph-mon[53211]: failed to encode map e80 with expected crc 2024-04-02T04:50:00.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:00 smithi122 ceph-mon[53211]: failed to encode map e80 with expected crc 2024-04-02T04:50:00.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:00 smithi122 ceph-mon[53211]: failed to encode map e80 with expected crc 2024-04-02T04:50:00.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:00 smithi122 ceph-mon[53211]: failed to encode map e80 with expected crc 2024-04-02T04:50:00.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:00 smithi122 ceph-mon[53211]: failed to encode map e80 with expected crc 2024-04-02T04:50:00.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:00 smithi122 ceph-mon[53211]: failed to encode map e80 with expected crc 2024-04-02T04:50:00.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:00 smithi122 ceph-mon[53211]: pgmap v75: 97 pgs: 80 active+clean, 5 active+undersized, 1 active+recovery_wait+degraded, 1 active+recovering+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+undersized+remapped; 349 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 5.9 MiB/s wr, 562 op/s; 103/1689 objects degraded (6.098%); 2.2 MiB/s, 26 objects/s recovering 2024-04-02T04:50:00.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:00 smithi122 ceph-mon[53211]: from='client.44151 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:50:00.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:00 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/2460546347' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:50:00.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:00 smithi122 ceph-mon[53211]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); Degraded data redundancy: 103/1689 objects degraded (6.098%), 9 pgs degraded 2024-04-02T04:50:00.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:00 smithi122 ceph-mon[53211]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-02T04:50:00.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:00 smithi122 ceph-mon[53211]: daemon grafana.smithi053 on smithi053 is in error state 2024-04-02T04:50:00.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:00 smithi122 ceph-mon[53211]: [WRN] PG_DEGRADED: Degraded data redundancy: 103/1689 objects degraded (6.098%), 9 pgs degraded 2024-04-02T04:50:00.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:00 smithi122 ceph-mon[53211]: pg 2.13 is active+undersized+degraded, acting [4,3] 2024-04-02T04:50:00.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:00 smithi122 ceph-mon[53211]: pg 2.1f is active+undersized+degraded, acting [7,4] 2024-04-02T04:50:00.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:00 smithi122 ceph-mon[53211]: pg 3.1 is active+undersized+degraded, acting [4,3] 2024-04-02T04:50:00.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:00 smithi122 ceph-mon[53211]: pg 3.7 is active+recovery_wait+degraded, acting [3,7,0] 2024-04-02T04:50:00.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:00 smithi122 ceph-mon[53211]: pg 3.b is active+recovering+degraded, acting [3,0,4] 2024-04-02T04:50:00.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:00 smithi122 ceph-mon[53211]: pg 3.12 is active+recovery_wait+undersized+degraded+remapped, acting [3,7] 2024-04-02T04:50:00.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:00 smithi122 ceph-mon[53211]: pg 3.17 is active+undersized+degraded, acting [5,3] 2024-04-02T04:50:00.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:00 smithi122 ceph-mon[53211]: pg 3.1b is active+undersized+degraded, acting [4,7] 2024-04-02T04:50:00.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:00 smithi122 ceph-mon[53211]: pg 3.1f is active+undersized+degraded, acting [5,2] 2024-04-02T04:50:00.398 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:00 smithi053 ceph-mon[67180]: failed to encode map e80 with expected crc 2024-04-02T04:50:00.398 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:00 smithi053 ceph-mon[67180]: failed to encode map e80 with expected crc 2024-04-02T04:50:00.398 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:00 smithi053 ceph-mon[67180]: failed to encode map e80 with expected crc 2024-04-02T04:50:00.398 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:00 smithi053 ceph-mon[67180]: failed to encode map e80 with expected crc 2024-04-02T04:50:00.398 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:00 smithi053 ceph-mon[67180]: failed to encode map e80 with expected crc 2024-04-02T04:50:00.398 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:00 smithi053 ceph-mon[67180]: failed to encode map e80 with expected crc 2024-04-02T04:50:00.398 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:00 smithi053 ceph-mon[67180]: pgmap v75: 97 pgs: 80 active+clean, 5 active+undersized, 1 active+recovery_wait+degraded, 1 active+recovering+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+undersized+remapped; 349 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 5.9 MiB/s wr, 562 op/s; 103/1689 objects degraded (6.098%); 2.2 MiB/s, 26 objects/s recovering 2024-04-02T04:50:00.399 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:00 smithi053 ceph-mon[67180]: from='client.44151 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:50:00.399 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:00 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/2460546347' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:50:00.399 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:00 smithi053 ceph-mon[67180]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); Degraded data redundancy: 103/1689 objects degraded (6.098%), 9 pgs degraded 2024-04-02T04:50:00.399 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:00 smithi053 ceph-mon[67180]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-02T04:50:00.399 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:00 smithi053 ceph-mon[67180]: daemon grafana.smithi053 on smithi053 is in error state 2024-04-02T04:50:00.399 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:00 smithi053 ceph-mon[67180]: [WRN] PG_DEGRADED: Degraded data redundancy: 103/1689 objects degraded (6.098%), 9 pgs degraded 2024-04-02T04:50:00.399 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:00 smithi053 ceph-mon[67180]: pg 2.13 is active+undersized+degraded, acting [4,3] 2024-04-02T04:50:00.399 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:00 smithi053 ceph-mon[67180]: pg 2.1f is active+undersized+degraded, acting [7,4] 2024-04-02T04:50:00.399 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:00 smithi053 ceph-mon[67180]: pg 3.1 is active+undersized+degraded, acting [4,3] 2024-04-02T04:50:00.399 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:00 smithi053 ceph-mon[67180]: pg 3.7 is active+recovery_wait+degraded, acting [3,7,0] 2024-04-02T04:50:00.399 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:00 smithi053 ceph-mon[67180]: pg 3.b is active+recovering+degraded, acting [3,0,4] 2024-04-02T04:50:00.399 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:00 smithi053 ceph-mon[67180]: pg 3.12 is active+recovery_wait+undersized+degraded+remapped, acting [3,7] 2024-04-02T04:50:00.399 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:00 smithi053 ceph-mon[67180]: pg 3.17 is active+undersized+degraded, acting [5,3] 2024-04-02T04:50:00.399 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:00 smithi053 ceph-mon[67180]: pg 3.1b is active+undersized+degraded, acting [4,7] 2024-04-02T04:50:00.400 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:00 smithi053 ceph-mon[67180]: pg 3.1f is active+undersized+degraded, acting [5,2] 2024-04-02T04:50:01.160 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:01 smithi053 ceph-mon[67180]: failed to encode map e81 with expected crc 2024-04-02T04:50:01.161 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:01 smithi053 ceph-mon[67180]: failed to encode map e81 with expected crc 2024-04-02T04:50:01.161 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:01 smithi053 ceph-mon[67180]: failed to encode map e81 with expected crc 2024-04-02T04:50:01.161 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:01 smithi053 ceph-mon[67180]: osdmap e81: 8 total, 8 up, 8 in 2024-04-02T04:50:01.161 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:01 smithi053 ceph-mon[67180]: failed to encode map e81 with expected crc 2024-04-02T04:50:01.161 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:01 smithi053 ceph-mon[67180]: failed to encode map e81 with expected crc 2024-04-02T04:50:01.161 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:01 smithi053 ceph-mon[67180]: Health check update: Degraded data redundancy: 103/1689 objects degraded (6.098%), 9 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:01.326 INFO:teuthology.orchestra.run.smithi053.stdout: 5 196786 10.88 MB/sec execute 311 sec latency 943.444 ms 2024-04-02T04:50:01.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:01 smithi122 ceph-mon[53211]: failed to encode map e81 with expected crc 2024-04-02T04:50:01.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:01 smithi122 ceph-mon[53211]: failed to encode map e81 with expected crc 2024-04-02T04:50:01.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:01 smithi122 ceph-mon[53211]: failed to encode map e81 with expected crc 2024-04-02T04:50:01.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:01 smithi122 ceph-mon[53211]: osdmap e81: 8 total, 8 up, 8 in 2024-04-02T04:50:01.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:01 smithi122 ceph-mon[53211]: failed to encode map e81 with expected crc 2024-04-02T04:50:01.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:01 smithi122 ceph-mon[53211]: failed to encode map e81 with expected crc 2024-04-02T04:50:01.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:01 smithi122 ceph-mon[53211]: Health check update: Degraded data redundancy: 103/1689 objects degraded (6.098%), 9 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:02.182 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:02 smithi053 ceph-mon[67180]: pgmap v77: 97 pgs: 92 active+clean, 1 active+recovery_wait+degraded, 1 active+recovering+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped; 345 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 5.7 MiB/s wr, 533 op/s; 42/1686 objects degraded (2.491%); 2.0 MiB/s, 40 objects/s recovering 2024-04-02T04:50:02.182 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:02 smithi053 ceph-mon[67180]: failed to encode map e82 with expected crc 2024-04-02T04:50:02.182 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:02 smithi053 ceph-mon[67180]: osdmap e82: 8 total, 8 up, 8 in 2024-04-02T04:50:02.182 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:02 smithi053 ceph-mon[67180]: failed to encode map e82 with expected crc 2024-04-02T04:50:02.182 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:02 smithi053 ceph-mon[67180]: failed to encode map e82 with expected crc 2024-04-02T04:50:02.182 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:02 smithi053 ceph-mon[67180]: failed to encode map e82 with expected crc 2024-04-02T04:50:02.182 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:02 smithi053 ceph-mon[67180]: failed to encode map e82 with expected crc 2024-04-02T04:50:02.326 INFO:teuthology.orchestra.run.smithi053.stdout: 5 197295 10.87 MB/sec execute 312 sec latency 29.567 ms 2024-04-02T04:50:02.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:02 smithi122 ceph-mon[53211]: pgmap v77: 97 pgs: 92 active+clean, 1 active+recovery_wait+degraded, 1 active+recovering+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped; 345 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 5.7 MiB/s wr, 533 op/s; 42/1686 objects degraded (2.491%); 2.0 MiB/s, 40 objects/s recovering 2024-04-02T04:50:02.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:02 smithi122 ceph-mon[53211]: failed to encode map e82 with expected crc 2024-04-02T04:50:02.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:02 smithi122 ceph-mon[53211]: osdmap e82: 8 total, 8 up, 8 in 2024-04-02T04:50:02.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:02 smithi122 ceph-mon[53211]: failed to encode map e82 with expected crc 2024-04-02T04:50:02.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:02 smithi122 ceph-mon[53211]: failed to encode map e82 with expected crc 2024-04-02T04:50:02.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:02 smithi122 ceph-mon[53211]: failed to encode map e82 with expected crc 2024-04-02T04:50:02.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:02 smithi122 ceph-mon[53211]: failed to encode map e82 with expected crc 2024-04-02T04:50:03.326 INFO:teuthology.orchestra.run.smithi053.stdout: 5 197610 10.87 MB/sec execute 313 sec latency 41.974 ms 2024-04-02T04:50:03.967 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:03 smithi053 ceph-mon[67180]: pgmap v79: 97 pgs: 96 active+clean, 1 active+recovery_wait+undersized+degraded+remapped; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 4.1 MiB/s wr, 335 op/s; 20/1596 objects degraded (1.253%); 6 B/s, 19 objects/s recovering 2024-04-02T04:50:03.967 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:03 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:03.967 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:03 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:03.967 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:03 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:50:03.967 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:03 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:50:03.967 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:03 smithi053 ceph-mon[67180]: pgmap v80: 97 pgs: 96 active+clean, 1 active+recovery_wait+undersized+degraded+remapped; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 4.8 MiB/s wr, 398 op/s; 20/1596 objects degraded (1.253%); 8 B/s, 22 objects/s recovering 2024-04-02T04:50:03.967 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:03 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:03.967 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:03 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:03.968 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:03 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:50:03.968 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:03 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:50:03.968 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:03 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:50:03.968 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:03 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:50:03.968 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:03 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-02T04:50:03.968 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:03 smithi053 ceph-mon[67180]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-02T04:50:03.968 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:03 smithi053 ceph-mon[67180]: Upgrade: osd.1 is safe to restart 2024-04-02T04:50:04.326 INFO:teuthology.orchestra.run.smithi053.stdout: 5 197983 10.89 MB/sec execute 314 sec latency 58.487 ms 2024-04-02T04:50:04.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:03 smithi122 ceph-mon[53211]: pgmap v79: 97 pgs: 96 active+clean, 1 active+recovery_wait+undersized+degraded+remapped; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 4.1 MiB/s wr, 335 op/s; 20/1596 objects degraded (1.253%); 6 B/s, 19 objects/s recovering 2024-04-02T04:50:04.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:03 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:04.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:03 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:04.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:03 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:50:04.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:03 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:50:04.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:03 smithi122 ceph-mon[53211]: pgmap v80: 97 pgs: 96 active+clean, 1 active+recovery_wait+undersized+degraded+remapped; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 4.8 MiB/s wr, 398 op/s; 20/1596 objects degraded (1.253%); 8 B/s, 22 objects/s recovering 2024-04-02T04:50:04.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:03 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:04.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:03 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:04.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:03 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:50:04.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:03 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:50:04.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:03 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:50:04.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:03 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:50:04.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:03 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-02T04:50:04.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:03 smithi122 ceph-mon[53211]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-02T04:50:04.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:03 smithi122 ceph-mon[53211]: Upgrade: osd.1 is safe to restart 2024-04-02T04:50:05.235 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:05 smithi053 ceph-mon[67180]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 20/1596 objects degraded (1.253%), 1 pg degraded) 2024-04-02T04:50:05.326 INFO:teuthology.orchestra.run.smithi053.stdout: 5 198383 10.88 MB/sec execute 315 sec latency 40.282 ms 2024-04-02T04:50:05.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:05 smithi122 ceph-mon[53211]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 20/1596 objects degraded (1.253%), 1 pg degraded) 2024-04-02T04:50:06.250 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:06 smithi053 ceph-mon[67180]: pgmap v81: 97 pgs: 97 active+clean; 340 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.7 MiB/s wr, 499 op/s; 5 B/s, 18 objects/s recovering 2024-04-02T04:50:06.250 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:06 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:06.250 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:06 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-02T04:50:06.250 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:06 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:50:06.326 INFO:teuthology.orchestra.run.smithi053.stdout: 5 198696 10.88 MB/sec execute 316 sec latency 39.918 ms 2024-04-02T04:50:06.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:06 smithi122 ceph-mon[53211]: pgmap v81: 97 pgs: 97 active+clean; 340 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.7 MiB/s wr, 499 op/s; 5 B/s, 18 objects/s recovering 2024-04-02T04:50:06.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:06 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:06.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:06 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-02T04:50:06.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:06 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:50:07.277 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:07 smithi053 ceph-mon[67180]: Upgrade: Updating osd.1 2024-04-02T04:50:07.277 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:07 smithi053 ceph-mon[67180]: Deploying daemon osd.1 on smithi053 2024-04-02T04:50:07.326 INFO:teuthology.orchestra.run.smithi053.stdout: 5 198873 10.86 MB/sec execute 317 sec latency 59.785 ms 2024-04-02T04:50:07.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:07 smithi122 ceph-mon[53211]: Upgrade: Updating osd.1 2024-04-02T04:50:07.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:07 smithi122 ceph-mon[53211]: Deploying daemon osd.1 on smithi053 2024-04-02T04:50:08.290 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:08 smithi053 ceph-mon[67180]: pgmap v82: 97 pgs: 97 active+clean; 340 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.0 MiB/s wr, 437 op/s; 0 B/s, 7 objects/s recovering 2024-04-02T04:50:08.326 INFO:teuthology.orchestra.run.smithi053.stdout: 5 199139 10.85 MB/sec execute 318 sec latency 42.469 ms 2024-04-02T04:50:08.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:08 smithi122 ceph-mon[53211]: pgmap v82: 97 pgs: 97 active+clean; 340 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.0 MiB/s wr, 437 op/s; 0 B/s, 7 objects/s recovering 2024-04-02T04:50:09.326 INFO:teuthology.orchestra.run.smithi053.stdout: 5 199446 10.83 MB/sec execute 319 sec latency 61.467 ms 2024-04-02T04:50:10.092 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:10 smithi053 ceph-mon[67180]: pgmap v83: 97 pgs: 97 active+clean; 343 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.0 MiB/s wr, 603 op/s; 0 B/s, 5 objects/s recovering 2024-04-02T04:50:10.326 INFO:teuthology.orchestra.run.smithi053.stdout: 5 199735 10.82 MB/sec execute 320 sec latency 59.225 ms 2024-04-02T04:50:10.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:10 smithi122 ceph-mon[53211]: pgmap v83: 97 pgs: 97 active+clean; 343 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.0 MiB/s wr, 603 op/s; 0 B/s, 5 objects/s recovering 2024-04-02T04:50:11.326 INFO:teuthology.orchestra.run.smithi053.stdout: 5 199970 10.81 MB/sec execute 321 sec latency 48.445 ms 2024-04-02T04:50:11.358 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:11 smithi053 systemd[1]: Stopping Ceph osd.1 for 48e16742-f0aa-11ee-b647-cb9ed24678a4... 2024-04-02T04:50:11.626 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:11 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1[39365]: 2024-04-02T04:50:11.486+0000 7f06bf064700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T04:50:11.626 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:11 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1[39365]: 2024-04-02T04:50:11.486+0000 7f06bf064700 -1 osd.1 82 *** Got signal Terminated *** 2024-04-02T04:50:11.626 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:11 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1[39365]: 2024-04-02T04:50:11.486+0000 7f06bf064700 -1 osd.1 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T04:50:12.136 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:12 smithi053 ceph-mon[67180]: pgmap v84: 97 pgs: 97 active+clean; 343 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.3 MiB/s wr, 517 op/s; 0 B/s, 4 objects/s recovering 2024-04-02T04:50:12.136 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:12 smithi053 ceph-mon[67180]: osd.1 marked itself down and dead 2024-04-02T04:50:12.136 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:12 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:12.136 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:12 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:50:12.136 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:12 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:12.326 INFO:teuthology.orchestra.run.smithi053.stdout: 5 200195 10.80 MB/sec execute 322 sec latency 248.605 ms 2024-04-02T04:50:12.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:12 smithi122 ceph-mon[53211]: pgmap v84: 97 pgs: 97 active+clean; 343 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.3 MiB/s wr, 517 op/s; 0 B/s, 4 objects/s recovering 2024-04-02T04:50:12.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:12 smithi122 ceph-mon[53211]: osd.1 marked itself down and dead 2024-04-02T04:50:12.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:12 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:12.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:12 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:50:12.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:12 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:13.057 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:12 smithi053 podman[75186]: 2024-04-02 04:50:12.735997142 +0000 UTC m=+1.424696619 container died ac1ad6266e63c0fbd8ed24f54191b7f74676adbd54d0c4891bc3a7063edb1c2a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1, build-date=2022-05-03T08:36:31.336870, vcs-type=git, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, version=8, name=centos-stream, ceph=True, RELEASE=HEAD, release=754, GIT_CLEAN=True, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=) 2024-04-02T04:50:13.322 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:13 smithi053 ceph-mon[67180]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-02T04:50:13.322 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:13 smithi053 ceph-mon[67180]: failed to encode map e83 with expected crc 2024-04-02T04:50:13.322 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:13 smithi053 ceph-mon[67180]: osdmap e83: 8 total, 7 up, 8 in 2024-04-02T04:50:13.322 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:13 smithi053 ceph-mon[67180]: failed to encode map e83 with expected crc 2024-04-02T04:50:13.322 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:13 smithi053 ceph-mon[67180]: failed to encode map e83 with expected crc 2024-04-02T04:50:13.322 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:13 smithi053 ceph-mon[67180]: failed to encode map e83 with expected crc 2024-04-02T04:50:13.322 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:13 smithi053 ceph-mon[67180]: failed to encode map e83 with expected crc 2024-04-02T04:50:13.322 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:13 smithi053 ceph-mon[67180]: failed to encode map e83 with expected crc 2024-04-02T04:50:13.322 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:13 smithi053 ceph-mon[67180]: failed to encode map e83 with expected crc 2024-04-02T04:50:13.326 INFO:teuthology.orchestra.run.smithi053.stdout: 5 200304 10.76 MB/sec execute 323 sec latency 1036.380 ms 2024-04-02T04:50:13.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:13 smithi122 ceph-mon[53211]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-02T04:50:13.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:13 smithi122 ceph-mon[53211]: failed to encode map e83 with expected crc 2024-04-02T04:50:13.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:13 smithi122 ceph-mon[53211]: osdmap e83: 8 total, 7 up, 8 in 2024-04-02T04:50:13.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:13 smithi122 ceph-mon[53211]: failed to encode map e83 with expected crc 2024-04-02T04:50:13.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:13 smithi122 ceph-mon[53211]: failed to encode map e83 with expected crc 2024-04-02T04:50:13.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:13 smithi122 ceph-mon[53211]: failed to encode map e83 with expected crc 2024-04-02T04:50:13.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:13 smithi122 ceph-mon[53211]: failed to encode map e83 with expected crc 2024-04-02T04:50:13.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:13 smithi122 ceph-mon[53211]: failed to encode map e83 with expected crc 2024-04-02T04:50:13.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:13 smithi122 ceph-mon[53211]: failed to encode map e83 with expected crc 2024-04-02T04:50:13.826 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:13 smithi053 podman[75186]: 2024-04-02 04:50:13.578689162 +0000 UTC m=+2.267388646 container cleanup ac1ad6266e63c0fbd8ed24f54191b7f74676adbd54d0c4891bc3a7063edb1c2a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, version=8, name=centos-stream, architecture=x86_64, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, distribution-scope=public, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, release=754) 2024-04-02T04:50:13.826 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:13 smithi053 bash[75186]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1 2024-04-02T04:50:14.081 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:13 smithi053 podman[75203]: 2024-04-02 04:50:13.931636484 +0000 UTC m=+1.195372461 container remove ac1ad6266e63c0fbd8ed24f54191b7f74676adbd54d0c4891bc3a7063edb1c2a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, maintainer=Guillaume Abrioux , name=centos-stream, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., vcs-type=git, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=) 2024-04-02T04:50:14.081 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:14 smithi053 ceph-mon[67180]: pgmap v86: 97 pgs: 25 peering, 72 active+clean; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.8 MiB/s wr, 535 op/s; 0 B/s, 1 objects/s recovering 2024-04-02T04:50:14.081 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:14 smithi053 ceph-mon[67180]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-02T04:50:14.081 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:14 smithi053 ceph-mon[67180]: failed to encode map e84 with expected crc 2024-04-02T04:50:14.081 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:14 smithi053 ceph-mon[67180]: osdmap e84: 8 total, 7 up, 8 in 2024-04-02T04:50:14.081 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:14 smithi053 ceph-mon[67180]: failed to encode map e84 with expected crc 2024-04-02T04:50:14.082 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:14 smithi053 ceph-mon[67180]: failed to encode map e84 with expected crc 2024-04-02T04:50:14.082 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:14 smithi053 ceph-mon[67180]: failed to encode map e84 with expected crc 2024-04-02T04:50:14.082 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:14 smithi053 ceph-mon[67180]: failed to encode map e84 with expected crc 2024-04-02T04:50:14.082 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:14 smithi053 ceph-mon[67180]: failed to encode map e84 with expected crc 2024-04-02T04:50:14.326 INFO:teuthology.orchestra.run.smithi053.stdout: 5 200682 10.75 MB/sec execute 324 sec latency 62.928 ms 2024-04-02T04:50:14.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:14 smithi122 ceph-mon[53211]: pgmap v86: 97 pgs: 25 peering, 72 active+clean; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.8 MiB/s wr, 535 op/s; 0 B/s, 1 objects/s recovering 2024-04-02T04:50:14.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:14 smithi122 ceph-mon[53211]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-02T04:50:14.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:14 smithi122 ceph-mon[53211]: failed to encode map e84 with expected crc 2024-04-02T04:50:14.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:14 smithi122 ceph-mon[53211]: osdmap e84: 8 total, 7 up, 8 in 2024-04-02T04:50:14.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:14 smithi122 ceph-mon[53211]: failed to encode map e84 with expected crc 2024-04-02T04:50:14.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:14 smithi122 ceph-mon[53211]: failed to encode map e84 with expected crc 2024-04-02T04:50:14.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:14 smithi122 ceph-mon[53211]: failed to encode map e84 with expected crc 2024-04-02T04:50:14.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:14 smithi122 ceph-mon[53211]: failed to encode map e84 with expected crc 2024-04-02T04:50:14.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:14 smithi122 ceph-mon[53211]: failed to encode map e84 with expected crc 2024-04-02T04:50:14.621 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:14 smithi053 podman[75289]: 2024-04-02 04:50:14.333127646 +0000 UTC m=+0.023193088 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:50:14.621 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:14 smithi053 podman[75289]: 2024-04-02 04:50:14.501169136 +0000 UTC m=+0.191234564 container create cf8d65b00772081fdfe2ee1d0a5b52706fae5ea92742e7d8b85ad8ea735b9677 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, RELEASE=squid-6cec793, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True) 2024-04-02T04:50:15.123 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:15 smithi053 ceph-mon[67180]: failed to encode map e84 with expected crc 2024-04-02T04:50:15.123 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:14 smithi053 podman[75289]: 2024-04-02 04:50:14.995925202 +0000 UTC m=+0.685990641 container init cf8d65b00772081fdfe2ee1d0a5b52706fae5ea92742e7d8b85ad8ea735b9677 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, ceph=True, RELEASE=squid-6cec793, io.buildah.version=1.33.5) 2024-04-02T04:50:15.123 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:15 smithi053 podman[75289]: 2024-04-02 04:50:15.000840612 +0000 UTC m=+0.690906051 container start cf8d65b00772081fdfe2ee1d0a5b52706fae5ea92742e7d8b85ad8ea735b9677 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-6cec793, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux ) 2024-04-02T04:50:15.326 INFO:teuthology.orchestra.run.smithi053.stdout: 5 200998 10.74 MB/sec execute 325 sec latency 44.196 ms 2024-04-02T04:50:15.373 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:15 smithi053 podman[75289]: 2024-04-02 04:50:15.182137394 +0000 UTC m=+0.872202826 container attach cf8d65b00772081fdfe2ee1d0a5b52706fae5ea92742e7d8b85ad8ea735b9677 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-6cec793, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-02T04:50:15.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:15 smithi122 ceph-mon[53211]: failed to encode map e84 with expected crc 2024-04-02T04:50:15.628 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:15 smithi053 podman[75289]: 2024-04-02 04:50:15.43999964 +0000 UTC m=+1.130065070 container died cf8d65b00772081fdfe2ee1d0a5b52706fae5ea92742e7d8b85ad8ea735b9677 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1-deactivate, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-6cec793, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-02T04:50:16.133 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:16 smithi053 ceph-mon[67180]: pgmap v88: 97 pgs: 2 active+undersized, 2 active+undersized+degraded, 10 stale+active+clean, 25 peering, 58 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 5.3 MiB/s wr, 514 op/s; 12/1539 objects degraded (0.780%) 2024-04-02T04:50:16.133 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:16 smithi053 ceph-mon[67180]: Health check failed: Degraded data redundancy: 12/1539 objects degraded (0.780%), 2 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:16.326 INFO:teuthology.orchestra.run.smithi053.stdout: 5 201645 10.79 MB/sec execute 326 sec latency 29.799 ms 2024-04-02T04:50:16.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:16 smithi122 ceph-mon[53211]: pgmap v88: 97 pgs: 2 active+undersized, 2 active+undersized+degraded, 10 stale+active+clean, 25 peering, 58 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 5.3 MiB/s wr, 514 op/s; 12/1539 objects degraded (0.780%) 2024-04-02T04:50:16.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:16 smithi122 ceph-mon[53211]: Health check failed: Degraded data redundancy: 12/1539 objects degraded (0.780%), 2 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:16.635 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:16 smithi053 podman[75314]: 2024-04-02 04:50:16.596911375 +0000 UTC m=+1.150550525 container remove cf8d65b00772081fdfe2ee1d0a5b52706fae5ea92742e7d8b85ad8ea735b9677 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-6cec793, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0) 2024-04-02T04:50:16.686 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:16 smithi053 systemd[1]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.1.service: Deactivated successfully. 2024-04-02T04:50:16.686 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:16 smithi053 systemd[1]: Stopped Ceph osd.1 for 48e16742-f0aa-11ee-b647-cb9ed24678a4. 2024-04-02T04:50:16.686 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:16 smithi053 systemd[1]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.1.service: Consumed 3min 32.085s CPU time. 2024-04-02T04:50:17.193 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:16 smithi053 systemd[1]: Starting Ceph osd.1 for 48e16742-f0aa-11ee-b647-cb9ed24678a4... 2024-04-02T04:50:17.326 INFO:teuthology.orchestra.run.smithi053.stdout: 5 202223 10.81 MB/sec execute 327 sec latency 40.792 ms 2024-04-02T04:50:17.447 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:17 smithi053 podman[75411]: 2024-04-02 04:50:17.143673482 +0000 UTC m=+0.023726257 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:50:17.448 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:17 smithi053 podman[75411]: 2024-04-02 04:50:17.292476488 +0000 UTC m=+0.172529239 container create 1d093d1c9e278386f5817caba11c714897e99bbc14245190381039910f0c82d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1-activate, org.label-schema.build-date=20240326, RELEASE=squid-6cec793, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T04:50:17.715 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:17 smithi053 podman[75411]: 2024-04-02 04:50:17.644107684 +0000 UTC m=+0.524160454 container init 1d093d1c9e278386f5817caba11c714897e99bbc14245190381039910f0c82d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1-activate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-6cec793, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T04:50:17.715 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:17 smithi053 podman[75411]: 2024-04-02 04:50:17.65049239 +0000 UTC m=+0.530545151 container start 1d093d1c9e278386f5817caba11c714897e99bbc14245190381039910f0c82d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1-activate, CEPH_POINT_RELEASE=, RELEASE=squid-6cec793, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T04:50:17.968 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:17 smithi053 podman[75411]: 2024-04-02 04:50:17.743628586 +0000 UTC m=+0.623681342 container attach 1d093d1c9e278386f5817caba11c714897e99bbc14245190381039910f0c82d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1-activate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-6cec793, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-02T04:50:18.224 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:18 smithi053 ceph-mon[67180]: pgmap v89: 97 pgs: 2 active+undersized, 2 active+undersized+degraded, 10 stale+active+clean, 25 peering, 58 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 2.5 MiB/s wr, 268 op/s; 12/1539 objects degraded (0.780%) 2024-04-02T04:50:18.224 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:18 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1-activate[75424]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-02T04:50:18.224 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:18 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1-activate[75424]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-02T04:50:18.225 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:18 smithi053 bash[75411]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-02T04:50:18.225 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:18 smithi053 bash[75411]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-02T04:50:18.225 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:18 smithi053 bash[75411]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-02T04:50:18.225 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:18 smithi053 bash[75411]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-02T04:50:18.225 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:18 smithi053 bash[75411]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-02T04:50:18.225 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:18 smithi053 bash[75411]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-02T04:50:18.225 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:18 smithi053 bash[75411]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-02T04:50:18.225 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:18 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1-activate[75424]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-02T04:50:18.225 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:18 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1-activate[75424]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-02T04:50:18.225 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:18 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1-activate[75424]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-02T04:50:18.225 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:18 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1-activate[75424]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-02T04:50:18.225 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:18 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1-activate[75424]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-02T04:50:18.326 INFO:teuthology.orchestra.run.smithi053.stdout: 5 202480 10.80 MB/sec execute 328 sec latency 63.852 ms 2024-04-02T04:50:18.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:18 smithi122 ceph-mon[53211]: pgmap v89: 97 pgs: 2 active+undersized, 2 active+undersized+degraded, 10 stale+active+clean, 25 peering, 58 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 2.5 MiB/s wr, 268 op/s; 12/1539 objects degraded (0.780%) 2024-04-02T04:50:18.474 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:18 smithi053 podman[75411]: 2024-04-02 04:50:18.23926256 +0000 UTC m=+1.119315315 container died 1d093d1c9e278386f5817caba11c714897e99bbc14245190381039910f0c82d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1-activate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240326, RELEASE=squid-6cec793, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-02T04:50:19.232 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:19 smithi053 ceph-mon[67180]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-02T04:50:19.327 INFO:teuthology.orchestra.run.smithi053.stdout: 5 202837 10.79 MB/sec execute 329 sec latency 42.750 ms 2024-04-02T04:50:19.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:19 smithi122 ceph-mon[53211]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-02T04:50:19.738 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:19 smithi053 podman[75556]: 2024-04-02 04:50:19.485847868 +0000 UTC m=+1.240598382 container remove 1d093d1c9e278386f5817caba11c714897e99bbc14245190381039910f0c82d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1-activate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-6cec793, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T04:50:19.990 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:19 smithi053 podman[75615]: 2024-04-02 04:50:19.662554913 +0000 UTC m=+0.033719315 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:50:19.990 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:19 smithi053 podman[75615]: 2024-04-02 04:50:19.851649314 +0000 UTC m=+0.222813703 container create e70e40949c2f34a2199cd5db9474ccfd4290ae038d7f3d781e86d139271fc92b (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-6cec793, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-02T04:50:20.246 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:20 smithi053 ceph-mon[67180]: pgmap v90: 97 pgs: 16 active+undersized, 23 active+undersized+degraded, 58 active+clean; 342 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.3 MiB/s wr, 579 op/s; 183/1710 objects degraded (10.702%); 0 B/s, 0 objects/s recovering 2024-04-02T04:50:20.246 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:20 smithi053 podman[75615]: 2024-04-02 04:50:20.224275771 +0000 UTC m=+0.595440155 container init e70e40949c2f34a2199cd5db9474ccfd4290ae038d7f3d781e86d139271fc92b (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, RELEASE=squid-6cec793) 2024-04-02T04:50:20.246 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:20 smithi053 podman[75615]: 2024-04-02 04:50:20.232827938 +0000 UTC m=+0.603992329 container start e70e40949c2f34a2199cd5db9474ccfd4290ae038d7f3d781e86d139271fc92b (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-6cec793) 2024-04-02T04:50:20.326 INFO:teuthology.orchestra.run.smithi053.stdout: 5 203241 10.78 MB/sec execute 330 sec latency 52.716 ms 2024-04-02T04:50:20.501 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:20 smithi053 ceph-osd[75634]: -- 172.21.15.53:0/2963989341 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x55a69d0d3ba0 con 0x55a69d0a5000 2024-04-02T04:50:20.501 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:20 smithi053 ceph-osd[75634]: -- 172.21.15.53:0/2963989341 <== mon.0 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a69de921e0 con 0x55a69d0a5000 2024-04-02T04:50:20.501 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:20 smithi053 bash[75615]: e70e40949c2f34a2199cd5db9474ccfd4290ae038d7f3d781e86d139271fc92b 2024-04-02T04:50:20.501 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:20 smithi053 systemd[1]: Started Ceph osd.1 for 48e16742-f0aa-11ee-b647-cb9ed24678a4. 2024-04-02T04:50:20.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:20 smithi122 ceph-mon[53211]: pgmap v90: 97 pgs: 16 active+undersized, 23 active+undersized+degraded, 58 active+clean; 342 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.3 MiB/s wr, 579 op/s; 183/1710 objects degraded (10.702%); 0 B/s, 0 objects/s recovering 2024-04-02T04:50:21.035 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:20 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1[75630]: 2024-04-02T04:50:20.809+0000 7f53a7e077c0 -1 Falling back to public interface 2024-04-02T04:50:21.327 INFO:teuthology.orchestra.run.smithi053.stdout: 5 203509 10.77 MB/sec execute 331 sec latency 49.880 ms 2024-04-02T04:50:21.547 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:21 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:21.547 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:21 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:21.547 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:21 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:50:21.547 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:21 smithi053 ceph-mon[67180]: Health check update: Degraded data redundancy: 183/1710 objects degraded (10.702%), 23 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:21.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:21 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:21.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:21 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:21.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:21 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:50:21.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:21 smithi122 ceph-mon[53211]: Health check update: Degraded data redundancy: 183/1710 objects degraded (10.702%), 23 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:22.327 INFO:teuthology.orchestra.run.smithi053.stdout: 5 203922 10.77 MB/sec execute 332 sec latency 40.678 ms 2024-04-02T04:50:22.485 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:22 smithi053 ceph-mon[67180]: pgmap v91: 97 pgs: 16 active+undersized, 23 active+undersized+degraded, 58 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.4 MiB/s wr, 543 op/s; 185/1722 objects degraded (10.743%); 0 B/s, 0 objects/s recovering 2024-04-02T04:50:22.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:22 smithi122 ceph-mon[53211]: pgmap v91: 97 pgs: 16 active+undersized, 23 active+undersized+degraded, 58 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.4 MiB/s wr, 543 op/s; 185/1722 objects degraded (10.743%); 0 B/s, 0 objects/s recovering 2024-04-02T04:50:23.327 INFO:teuthology.orchestra.run.smithi053.stdout: 5 204376 10.75 MB/sec execute 333 sec latency 50.511 ms 2024-04-02T04:50:24.327 INFO:teuthology.orchestra.run.smithi053.stdout: 5 204629 10.75 MB/sec execute 334 sec latency 43.173 ms 2024-04-02T04:50:24.591 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:24 smithi053 ceph-mon[67180]: pgmap v92: 97 pgs: 16 active+undersized, 23 active+undersized+degraded, 58 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.4 MiB/s wr, 544 op/s; 167/1617 objects degraded (10.328%); 0 B/s, 0 objects/s recovering 2024-04-02T04:50:24.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:24 smithi122 ceph-mon[53211]: pgmap v92: 97 pgs: 16 active+undersized, 23 active+undersized+degraded, 58 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.4 MiB/s wr, 544 op/s; 167/1617 objects degraded (10.328%); 0 B/s, 0 objects/s recovering 2024-04-02T04:50:25.327 INFO:teuthology.orchestra.run.smithi053.stdout: 5 205009 10.77 MB/sec execute 335 sec latency 66.542 ms 2024-04-02T04:50:25.863 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:25 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1[75630]: 2024-04-02T04:50:25.627+0000 7f53a7e077c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-04-02T04:50:26.327 INFO:teuthology.orchestra.run.smithi053.stdout: 5 205402 10.77 MB/sec execute 336 sec latency 62.419 ms 2024-04-02T04:50:26.643 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:26 smithi053 ceph-mon[67180]: pgmap v93: 97 pgs: 16 active+undersized, 23 active+undersized+degraded, 58 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.8 MiB/s wr, 474 op/s; 166/1611 objects degraded (10.304%); 0 B/s, 0 objects/s recovering 2024-04-02T04:50:26.643 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:26 smithi053 ceph-mon[67180]: Health check update: Degraded data redundancy: 166/1611 objects degraded (10.304%), 23 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:26.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:26 smithi122 ceph-mon[53211]: pgmap v93: 97 pgs: 16 active+undersized, 23 active+undersized+degraded, 58 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.8 MiB/s wr, 474 op/s; 166/1611 objects degraded (10.304%); 0 B/s, 0 objects/s recovering 2024-04-02T04:50:26.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:26 smithi122 ceph-mon[53211]: Health check update: Degraded data redundancy: 166/1611 objects degraded (10.304%), 23 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:27.155 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:26 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1[75630]: 2024-04-02T04:50:26.898+0000 7f53a7e077c0 -1 osd.1 82 log_to_monitors true 2024-04-02T04:50:27.327 INFO:teuthology.orchestra.run.smithi053.stdout: 5 205711 10.77 MB/sec execute 337 sec latency 45.503 ms 2024-04-02T04:50:27.660 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:27 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:27.660 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:27 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:50:27.660 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:27 smithi053 ceph-mon[67180]: from='osd.1 [v2:172.21.15.53:6810/2846241084,v1:172.21.15.53:6811/2846241084]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-02T04:50:27.660 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:27 smithi053 ceph-mon[67180]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-02T04:50:27.660 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:27 smithi053 ceph-mon[67180]: pgmap v94: 97 pgs: 16 active+undersized, 23 active+undersized+degraded, 58 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.6 MiB/s wr, 456 op/s; 166/1611 objects degraded (10.304%); 0 B/s, 0 objects/s recovering 2024-04-02T04:50:27.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:27 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:27.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:27 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:50:27.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:27 smithi122 ceph-mon[53211]: from='osd.1 [v2:172.21.15.53:6810/2846241084,v1:172.21.15.53:6811/2846241084]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-02T04:50:27.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:27 smithi122 ceph-mon[53211]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-02T04:50:27.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:27 smithi122 ceph-mon[53211]: pgmap v94: 97 pgs: 16 active+undersized, 23 active+undersized+degraded, 58 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.6 MiB/s wr, 456 op/s; 166/1611 objects degraded (10.304%); 0 B/s, 0 objects/s recovering 2024-04-02T04:50:28.188 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:50:27 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1[75630]: 2024-04-02T04:50:27.985+0000 7f5395257700 -1 osd.1 82 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T04:50:28.327 INFO:teuthology.orchestra.run.smithi053.stdout: 5 205909 10.75 MB/sec execute 338 sec latency 62.092 ms 2024-04-02T04:50:28.720 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:28 smithi053 ceph-mon[67180]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-02T04:50:28.721 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:28 smithi053 ceph-mon[67180]: failed to encode map e85 with expected crc 2024-04-02T04:50:28.721 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:28 smithi053 ceph-mon[67180]: osdmap e85: 8 total, 7 up, 8 in 2024-04-02T04:50:28.721 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:28 smithi053 ceph-mon[67180]: from='osd.1 [v2:172.21.15.53:6810/2846241084,v1:172.21.15.53:6811/2846241084]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-02T04:50:28.721 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:28 smithi053 ceph-mon[67180]: failed to encode map e85 with expected crc 2024-04-02T04:50:28.721 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:28 smithi053 ceph-mon[67180]: failed to encode map e85 with expected crc 2024-04-02T04:50:28.721 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:28 smithi053 ceph-mon[67180]: failed to encode map e85 with expected crc 2024-04-02T04:50:28.721 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:28 smithi053 ceph-mon[67180]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-02T04:50:28.721 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:28 smithi053 ceph-mon[67180]: failed to encode map e85 with expected crc 2024-04-02T04:50:28.721 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:28 smithi053 ceph-mon[67180]: failed to encode map e85 with expected crc 2024-04-02T04:50:28.721 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:28 smithi053 ceph-mon[67180]: from='osd.1 ' entity='osd.1' 2024-04-02T04:50:28.721 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:28 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:28.721 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:28 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:28.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:28 smithi122 ceph-mon[53211]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-02T04:50:28.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:28 smithi122 ceph-mon[53211]: failed to encode map e85 with expected crc 2024-04-02T04:50:28.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:28 smithi122 ceph-mon[53211]: osdmap e85: 8 total, 7 up, 8 in 2024-04-02T04:50:28.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:28 smithi122 ceph-mon[53211]: from='osd.1 [v2:172.21.15.53:6810/2846241084,v1:172.21.15.53:6811/2846241084]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-02T04:50:28.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:28 smithi122 ceph-mon[53211]: failed to encode map e85 with expected crc 2024-04-02T04:50:28.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:28 smithi122 ceph-mon[53211]: failed to encode map e85 with expected crc 2024-04-02T04:50:28.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:28 smithi122 ceph-mon[53211]: failed to encode map e85 with expected crc 2024-04-02T04:50:28.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:28 smithi122 ceph-mon[53211]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-02T04:50:28.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:28 smithi122 ceph-mon[53211]: failed to encode map e85 with expected crc 2024-04-02T04:50:28.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:28 smithi122 ceph-mon[53211]: failed to encode map e85 with expected crc 2024-04-02T04:50:28.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:28 smithi122 ceph-mon[53211]: from='osd.1 ' entity='osd.1' 2024-04-02T04:50:28.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:28 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:28.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:28 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:29.327 INFO:teuthology.orchestra.run.smithi053.stdout: 5 206103 10.73 MB/sec execute 339 sec latency 346.968 ms 2024-04-02T04:50:29.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:29 smithi053 ceph-mon[67180]: pgmap v96: 97 pgs: 16 active+undersized, 23 active+undersized+degraded, 58 active+clean; 348 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.4 MiB/s wr, 520 op/s; 177/1683 objects degraded (10.517%) 2024-04-02T04:50:29.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:29 smithi053 ceph-mon[67180]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-02T04:50:29.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:29 smithi053 ceph-mon[67180]: failed to encode map e86 with expected crc 2024-04-02T04:50:29.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:29 smithi053 ceph-mon[67180]: osd.1 [v2:172.21.15.53:6810/2846241084,v1:172.21.15.53:6811/2846241084] boot 2024-04-02T04:50:29.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:29 smithi053 ceph-mon[67180]: osdmap e86: 8 total, 8 up, 8 in 2024-04-02T04:50:29.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:29 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:50:29.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:29 smithi053 ceph-mon[67180]: failed to encode map e86 with expected crc 2024-04-02T04:50:30.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:29 smithi122 ceph-mon[53211]: pgmap v96: 97 pgs: 16 active+undersized, 23 active+undersized+degraded, 58 active+clean; 348 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.4 MiB/s wr, 520 op/s; 177/1683 objects degraded (10.517%) 2024-04-02T04:50:30.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:29 smithi122 ceph-mon[53211]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-02T04:50:30.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:29 smithi122 ceph-mon[53211]: failed to encode map e86 with expected crc 2024-04-02T04:50:30.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:29 smithi122 ceph-mon[53211]: osd.1 [v2:172.21.15.53:6810/2846241084,v1:172.21.15.53:6811/2846241084] boot 2024-04-02T04:50:30.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:29 smithi122 ceph-mon[53211]: osdmap e86: 8 total, 8 up, 8 in 2024-04-02T04:50:30.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:29 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T04:50:30.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:29 smithi122 ceph-mon[53211]: failed to encode map e86 with expected crc 2024-04-02T04:50:30.291 INFO:teuthology.orchestra.run.smithi053.stdout:true 2024-04-02T04:50:30.327 INFO:teuthology.orchestra.run.smithi053.stdout: 5 206194 10.70 MB/sec execute 340 sec latency 1340.928 ms 2024-04-02T04:50:30.894 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:30 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:30.894 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:30 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:30.894 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:30 smithi053 ceph-mon[67180]: failed to encode map e87 with expected crc 2024-04-02T04:50:30.894 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:30 smithi053 ceph-mon[67180]: failed to encode map e87 with expected crc 2024-04-02T04:50:30.894 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:30 smithi053 ceph-mon[67180]: failed to encode map e87 with expected crc 2024-04-02T04:50:30.894 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:30 smithi053 ceph-mon[67180]: failed to encode map e87 with expected crc 2024-04-02T04:50:30.894 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:30 smithi053 ceph-mon[67180]: failed to encode map e87 with expected crc 2024-04-02T04:50:30.894 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:30 smithi053 ceph-mon[67180]: failed to encode map e87 with expected crc 2024-04-02T04:50:30.894 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:30 smithi053 ceph-mon[67180]: failed to encode map e87 with expected crc 2024-04-02T04:50:30.894 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:30 smithi053 ceph-mon[67180]: failed to encode map e87 with expected crc 2024-04-02T04:50:30.895 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:30 smithi053 ceph-mon[67180]: failed to encode map e87 with expected crc 2024-04-02T04:50:30.895 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:30 smithi053 ceph-mon[67180]: osdmap e87: 8 total, 8 up, 8 in 2024-04-02T04:50:30.895 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:30 smithi053 ceph-mon[67180]: failed to encode map e87 with expected crc 2024-04-02T04:50:30.895 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:30 smithi053 ceph-mon[67180]: failed to encode map e87 with expected crc 2024-04-02T04:50:30.895 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:30 smithi053 ceph-mon[67180]: failed to encode map e87 with expected crc 2024-04-02T04:50:30.895 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:30 smithi053 ceph-mon[67180]: failed to encode map e87 with expected crc 2024-04-02T04:50:30.895 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:30 smithi053 ceph-mon[67180]: from='client.34232 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:50:30.895 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:30 smithi053 ceph-mon[67180]: Health check update: Degraded data redundancy: 177/1683 objects degraded (10.517%), 23 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:31.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:30 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:31.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:30 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:31.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:30 smithi122 ceph-mon[53211]: failed to encode map e87 with expected crc 2024-04-02T04:50:31.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:30 smithi122 ceph-mon[53211]: failed to encode map e87 with expected crc 2024-04-02T04:50:31.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:30 smithi122 ceph-mon[53211]: failed to encode map e87 with expected crc 2024-04-02T04:50:31.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:30 smithi122 ceph-mon[53211]: failed to encode map e87 with expected crc 2024-04-02T04:50:31.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:30 smithi122 ceph-mon[53211]: failed to encode map e87 with expected crc 2024-04-02T04:50:31.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:30 smithi122 ceph-mon[53211]: failed to encode map e87 with expected crc 2024-04-02T04:50:31.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:30 smithi122 ceph-mon[53211]: failed to encode map e87 with expected crc 2024-04-02T04:50:31.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:30 smithi122 ceph-mon[53211]: failed to encode map e87 with expected crc 2024-04-02T04:50:31.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:30 smithi122 ceph-mon[53211]: failed to encode map e87 with expected crc 2024-04-02T04:50:31.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:30 smithi122 ceph-mon[53211]: osdmap e87: 8 total, 8 up, 8 in 2024-04-02T04:50:31.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:30 smithi122 ceph-mon[53211]: failed to encode map e87 with expected crc 2024-04-02T04:50:31.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:30 smithi122 ceph-mon[53211]: failed to encode map e87 with expected crc 2024-04-02T04:50:31.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:30 smithi122 ceph-mon[53211]: failed to encode map e87 with expected crc 2024-04-02T04:50:31.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:30 smithi122 ceph-mon[53211]: failed to encode map e87 with expected crc 2024-04-02T04:50:31.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:30 smithi122 ceph-mon[53211]: from='client.34232 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:50:31.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:30 smithi122 ceph-mon[53211]: Health check update: Degraded data redundancy: 177/1683 objects degraded (10.517%), 23 pgs degraded (PG_DEGRADED) 2024-04-02T04:50:31.167 INFO:teuthology.orchestra.run.smithi053.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-02T04:50:31.167 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager.smithi053 smithi053 *:9093,9094 running (5m) 2s ago 14m 24.3M - 0.23.0 ba2b418f427c bf50c1cd2b3d 2024-04-02T04:50:31.168 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi053 smithi053 running (82s) 2s ago 14m 6660k - 19.0.0-1979-g6cec793d 78f830190839 4ce8915e0157 2024-04-02T04:50:31.168 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi122 smithi122 running (76s) 71s ago 13m 6660k - 19.0.0-1979-g6cec793d 78f830190839 e2b5369e6fdf 2024-04-02T04:50:31.168 INFO:teuthology.orchestra.run.smithi053.stdout:grafana.smithi053 smithi053 *:3000 error 2s ago 13m - - 2024-04-02T04:50:31.168 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi053.nfrfot smithi053 running (8m) 2s ago 8m 37.3M - 17.2.0 e1d6a67b021e c94e9e643514 2024-04-02T04:50:31.168 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi122.mwoyoh smithi122 running (8m) 71s ago 8m 15.6M - 17.2.0 e1d6a67b021e 73376d9ee4b8 2024-04-02T04:50:31.168 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi053.legdmv smithi053 *:8443,9283,8765 running (4m) 2s ago 15m 503M - 19.0.0-1979-g6cec793d 78f830190839 c12d516a31bc 2024-04-02T04:50:31.168 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi122.cpwlyk smithi122 *:8443,9283,8765 running (3m) 71s ago 13m 449M - 19.0.0-1979-g6cec793d 78f830190839 9a545c5e7d81 2024-04-02T04:50:31.168 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi053 smithi053 running (2m) 2s ago 15m 49.4M 2048M 19.0.0-1979-g6cec793d 78f830190839 f53a44f19c00 2024-04-02T04:50:31.168 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi122 smithi122 running (2m) 71s ago 13m 34.8M 2048M 19.0.0-1979-g6cec793d 78f830190839 44b200fed3c5 2024-04-02T04:50:31.168 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.0.0.smithi053.mhkdcb smithi053 *:2049 running (7m) 2s ago 7m 287M - 4.0 e1d6a67b021e 07e5da7bd2c5 2024-04-02T04:50:31.168 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.1.0.smithi122.mauvsb smithi122 *:2049 running (7m) 71s ago 7m 81.7M - 4.0 e1d6a67b021e 4f866685005e 2024-04-02T04:50:31.168 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi053 smithi053 *:9100 running (4m) 2s ago 13m 18.9M - 1.3.1 1dbe0e931976 773b71971e67 2024-04-02T04:50:31.168 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi122 smithi122 *:9100 running (4m) 71s ago 13m 15.7M - 1.3.1 1dbe0e931976 475d439c7980 2024-04-02T04:50:31.169 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 smithi053 running (45s) 2s ago 12m 235M 1937M 19.0.0-1979-g6cec793d 78f830190839 490c346c7639 2024-04-02T04:50:31.169 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 smithi053 running (11s) 2s ago 12m 12.1M 1937M 19.0.0-1979-g6cec793d 78f830190839 e70e40949c2f 2024-04-02T04:50:31.169 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 smithi053 running (11m) 2s ago 11m 537M 1937M 17.2.0 e1d6a67b021e 86964c8a94ff 2024-04-02T04:50:31.169 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 smithi053 running (11m) 2s ago 11m 367M 1937M 17.2.0 e1d6a67b021e 7e8b9e98ae2b 2024-04-02T04:50:31.169 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 smithi122 running (10m) 71s ago 10m 557M 2704M 17.2.0 e1d6a67b021e 46076afcd7bc 2024-04-02T04:50:31.169 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 smithi122 running (10m) 71s ago 10m 389M 2704M 17.2.0 e1d6a67b021e e687a2f6da4d 2024-04-02T04:50:31.169 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 smithi122 running (10m) 71s ago 10m 568M 2704M 17.2.0 e1d6a67b021e 9e371df9a839 2024-04-02T04:50:31.169 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 smithi122 running (9m) 71s ago 9m 480M 2704M 17.2.0 e1d6a67b021e 9b3cc398254d 2024-04-02T04:50:31.169 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus.smithi053 smithi053 *:9095 running (3m) 2s ago 12m 67.4M - 2.33.4 514e6a882f6e 1dd55399da10 2024-04-02T04:50:31.328 INFO:teuthology.orchestra.run.smithi053.stdout: 5 206337 10.68 MB/sec execute 341 sec latency 2106.665 ms 2024-04-02T04:50:31.763 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:50:31.764 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": { 2024-04-02T04:50:31.764 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:50:31.764 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:50:31.764 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": { 2024-04-02T04:50:31.764 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:50:31.764 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:50:31.764 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": { 2024-04-02T04:50:31.764 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-04-02T04:50:31.764 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:50:31.764 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:50:31.764 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": { 2024-04-02T04:50:31.764 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:50:31.764 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:50:31.764 INFO:teuthology.orchestra.run.smithi053.stdout: "overall": { 2024-04-02T04:50:31.765 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-04-02T04:50:31.765 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 6 2024-04-02T04:50:31.765 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-02T04:50:31.765 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:50:32.151 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:31 smithi053 ceph-mon[67180]: from='client.34236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:50:32.151 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:31 smithi053 ceph-mon[67180]: pgmap v99: 97 pgs: 3 peering, 14 active+undersized, 22 active+undersized+degraded, 58 active+clean; 348 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.6 MiB/s wr, 408 op/s; 167/1680 objects degraded (9.940%) 2024-04-02T04:50:32.151 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:31 smithi053 ceph-mon[67180]: failed to encode map e88 with expected crc 2024-04-02T04:50:32.151 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:31 smithi053 ceph-mon[67180]: failed to encode map e88 with expected crc 2024-04-02T04:50:32.151 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:31 smithi053 ceph-mon[67180]: failed to encode map e88 with expected crc 2024-04-02T04:50:32.152 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:31 smithi053 ceph-mon[67180]: failed to encode map e88 with expected crc 2024-04-02T04:50:32.152 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:31 smithi053 ceph-mon[67180]: failed to encode map e88 with expected crc 2024-04-02T04:50:32.152 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:31 smithi053 ceph-mon[67180]: failed to encode map e88 with expected crc 2024-04-02T04:50:32.152 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:31 smithi053 ceph-mon[67180]: osdmap e88: 8 total, 8 up, 8 in 2024-04-02T04:50:32.152 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:31 smithi053 ceph-mon[67180]: failed to encode map e88 with expected crc 2024-04-02T04:50:32.152 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:31 smithi053 ceph-mon[67180]: failed to encode map e88 with expected crc 2024-04-02T04:50:32.152 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:31 smithi053 ceph-mon[67180]: failed to encode map e88 with expected crc 2024-04-02T04:50:32.152 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:31 smithi053 ceph-mon[67180]: failed to encode map e88 with expected crc 2024-04-02T04:50:32.152 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:31 smithi053 ceph-mon[67180]: failed to encode map e88 with expected crc 2024-04-02T04:50:32.152 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:31 smithi053 ceph-mon[67180]: failed to encode map e88 with expected crc 2024-04-02T04:50:32.152 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:31 smithi053 ceph-mon[67180]: failed to encode map e88 with expected crc 2024-04-02T04:50:32.152 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:31 smithi053 ceph-mon[67180]: failed to encode map e88 with expected crc 2024-04-02T04:50:32.152 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:31 smithi053 ceph-mon[67180]: from='client.34240 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:50:32.152 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:31 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/1921147828' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:50:32.273 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:50:32.274 INFO:teuthology.orchestra.run.smithi053.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae", 2024-04-02T04:50:32.274 INFO:teuthology.orchestra.run.smithi053.stdout: "in_progress": true, 2024-04-02T04:50:32.274 INFO:teuthology.orchestra.run.smithi053.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-02T04:50:32.274 INFO:teuthology.orchestra.run.smithi053.stdout: "services_complete": [ 2024-04-02T04:50:32.274 INFO:teuthology.orchestra.run.smithi053.stdout: "mon", 2024-04-02T04:50:32.274 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr", 2024-04-02T04:50:32.274 INFO:teuthology.orchestra.run.smithi053.stdout: "crash" 2024-04-02T04:50:32.274 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-04-02T04:50:32.274 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": "8/23 daemons upgraded", 2024-04-02T04:50:32.274 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "Currently upgrading osd daemons", 2024-04-02T04:50:32.275 INFO:teuthology.orchestra.run.smithi053.stdout: "is_paused": false 2024-04-02T04:50:32.275 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:50:32.327 INFO:teuthology.orchestra.run.smithi053.stdout: 5 206724 10.68 MB/sec execute 342 sec latency 38.915 ms 2024-04-02T04:50:32.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:31 smithi122 ceph-mon[53211]: from='client.34236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:50:32.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:31 smithi122 ceph-mon[53211]: pgmap v99: 97 pgs: 3 peering, 14 active+undersized, 22 active+undersized+degraded, 58 active+clean; 348 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.6 MiB/s wr, 408 op/s; 167/1680 objects degraded (9.940%) 2024-04-02T04:50:32.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:31 smithi122 ceph-mon[53211]: failed to encode map e88 with expected crc 2024-04-02T04:50:32.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:31 smithi122 ceph-mon[53211]: failed to encode map e88 with expected crc 2024-04-02T04:50:32.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:31 smithi122 ceph-mon[53211]: failed to encode map e88 with expected crc 2024-04-02T04:50:32.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:31 smithi122 ceph-mon[53211]: failed to encode map e88 with expected crc 2024-04-02T04:50:32.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:31 smithi122 ceph-mon[53211]: failed to encode map e88 with expected crc 2024-04-02T04:50:32.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:31 smithi122 ceph-mon[53211]: failed to encode map e88 with expected crc 2024-04-02T04:50:32.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:31 smithi122 ceph-mon[53211]: osdmap e88: 8 total, 8 up, 8 in 2024-04-02T04:50:32.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:31 smithi122 ceph-mon[53211]: failed to encode map e88 with expected crc 2024-04-02T04:50:32.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:31 smithi122 ceph-mon[53211]: failed to encode map e88 with expected crc 2024-04-02T04:50:32.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:31 smithi122 ceph-mon[53211]: failed to encode map e88 with expected crc 2024-04-02T04:50:32.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:31 smithi122 ceph-mon[53211]: failed to encode map e88 with expected crc 2024-04-02T04:50:32.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:31 smithi122 ceph-mon[53211]: failed to encode map e88 with expected crc 2024-04-02T04:50:32.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:31 smithi122 ceph-mon[53211]: failed to encode map e88 with expected crc 2024-04-02T04:50:32.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:31 smithi122 ceph-mon[53211]: failed to encode map e88 with expected crc 2024-04-02T04:50:32.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:31 smithi122 ceph-mon[53211]: failed to encode map e88 with expected crc 2024-04-02T04:50:32.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:31 smithi122 ceph-mon[53211]: from='client.34240 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:50:32.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:31 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/1921147828' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:50:32.811 INFO:teuthology.orchestra.run.smithi053.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Degraded data redundancy: 167/1680 objects degraded (9.940%), 22 pgs degraded 2024-04-02T04:50:32.811 INFO:teuthology.orchestra.run.smithi053.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-02T04:50:32.811 INFO:teuthology.orchestra.run.smithi053.stdout: daemon grafana.smithi053 on smithi053 is in error state 2024-04-02T04:50:32.811 INFO:teuthology.orchestra.run.smithi053.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 167/1680 objects degraded (9.940%), 22 pgs degraded 2024-04-02T04:50:32.811 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.0 is active+undersized+degraded, acting [7,0] 2024-04-02T04:50:32.811 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.2 is active+undersized+degraded, acting [5,6] 2024-04-02T04:50:32.811 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.4 is active+undersized+degraded, acting [0,7] 2024-04-02T04:50:32.812 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.6 is active+undersized+degraded, acting [6,4] 2024-04-02T04:50:32.812 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.8 is active+undersized+degraded, acting [7,0] 2024-04-02T04:50:32.812 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.9 is active+undersized+degraded, acting [7,3] 2024-04-02T04:50:32.812 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.a is active+undersized+degraded, acting [3,7] 2024-04-02T04:50:32.812 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.d is active+undersized+degraded, acting [4,3] 2024-04-02T04:50:32.812 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.10 is active+undersized+degraded, acting [2,0] 2024-04-02T04:50:32.812 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.11 is active+undersized+degraded, acting [6,4] 2024-04-02T04:50:32.812 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.15 is active+undersized+degraded, acting [5,0] 2024-04-02T04:50:32.812 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.0 is active+undersized+degraded, acting [2,6] 2024-04-02T04:50:32.812 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.4 is active+undersized+degraded, acting [2,5] 2024-04-02T04:50:32.812 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.8 is active+undersized+degraded, acting [3,7] 2024-04-02T04:50:32.812 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.a is active+undersized+degraded, acting [6,4] 2024-04-02T04:50:32.812 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.e is active+undersized+degraded, acting [7,4] 2024-04-02T04:50:32.812 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.16 is active+undersized+degraded, acting [5,7] 2024-04-02T04:50:32.813 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.18 is active+undersized+degraded, acting [3,0] 2024-04-02T04:50:32.813 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.19 is active+undersized+degraded, acting [3,4] 2024-04-02T04:50:32.813 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.1a is active+undersized+degraded, acting [4,2] 2024-04-02T04:50:32.813 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.1c is active+undersized+degraded, acting [5,4] 2024-04-02T04:50:32.813 INFO:teuthology.orchestra.run.smithi053.stdout: pg 4.14 is active+undersized+degraded, acting [3,7] 2024-04-02T04:50:33.063 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:32 smithi053 ceph-mon[67180]: failed to encode map e89 with expected crc 2024-04-02T04:50:33.063 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:32 smithi053 ceph-mon[67180]: osdmap e89: 8 total, 8 up, 8 in 2024-04-02T04:50:33.063 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:32 smithi053 ceph-mon[67180]: failed to encode map e89 with expected crc 2024-04-02T04:50:33.063 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:32 smithi053 ceph-mon[67180]: failed to encode map e89 with expected crc 2024-04-02T04:50:33.063 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:32 smithi053 ceph-mon[67180]: failed to encode map e89 with expected crc 2024-04-02T04:50:33.063 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:32 smithi053 ceph-mon[67180]: failed to encode map e89 with expected crc 2024-04-02T04:50:33.063 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:32 smithi053 ceph-mon[67180]: failed to encode map e89 with expected crc 2024-04-02T04:50:33.063 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:32 smithi053 ceph-mon[67180]: failed to encode map e89 with expected crc 2024-04-02T04:50:33.063 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:32 smithi053 ceph-mon[67180]: failed to encode map e89 with expected crc 2024-04-02T04:50:33.063 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:32 smithi053 ceph-mon[67180]: from='client.44173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:50:33.064 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:32 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/3320961623' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:50:33.327 INFO:teuthology.orchestra.run.smithi053.stdout: 5 206985 10.67 MB/sec execute 343 sec latency 649.858 ms 2024-04-02T04:50:33.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:32 smithi122 ceph-mon[53211]: failed to encode map e89 with expected crc 2024-04-02T04:50:33.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:32 smithi122 ceph-mon[53211]: osdmap e89: 8 total, 8 up, 8 in 2024-04-02T04:50:33.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:32 smithi122 ceph-mon[53211]: failed to encode map e89 with expected crc 2024-04-02T04:50:33.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:32 smithi122 ceph-mon[53211]: failed to encode map e89 with expected crc 2024-04-02T04:50:33.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:32 smithi122 ceph-mon[53211]: failed to encode map e89 with expected crc 2024-04-02T04:50:33.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:32 smithi122 ceph-mon[53211]: failed to encode map e89 with expected crc 2024-04-02T04:50:33.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:32 smithi122 ceph-mon[53211]: failed to encode map e89 with expected crc 2024-04-02T04:50:33.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:32 smithi122 ceph-mon[53211]: failed to encode map e89 with expected crc 2024-04-02T04:50:33.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:32 smithi122 ceph-mon[53211]: failed to encode map e89 with expected crc 2024-04-02T04:50:33.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:32 smithi122 ceph-mon[53211]: from='client.44173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:50:33.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:32 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/3320961623' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:50:34.247 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:33 smithi053 ceph-mon[67180]: pgmap v102: 97 pgs: 6 peering, 3 active+undersized+remapped, 88 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 4.2 MiB/s wr, 429 op/s; 1024 KiB/s, 74 keys/s, 52 objects/s recovering 2024-04-02T04:50:34.247 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:33 smithi053 ceph-mon[67180]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 167/1680 objects degraded (9.940%), 22 pgs degraded) 2024-04-02T04:50:34.247 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:33 smithi053 ceph-mon[67180]: failed to encode map e90 with expected crc 2024-04-02T04:50:34.247 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:33 smithi053 ceph-mon[67180]: osdmap e90: 8 total, 8 up, 8 in 2024-04-02T04:50:34.247 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:33 smithi053 ceph-mon[67180]: failed to encode map e90 with expected crc 2024-04-02T04:50:34.247 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:33 smithi053 ceph-mon[67180]: failed to encode map e90 with expected crc 2024-04-02T04:50:34.247 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:33 smithi053 ceph-mon[67180]: failed to encode map e90 with expected crc 2024-04-02T04:50:34.247 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:33 smithi053 ceph-mon[67180]: failed to encode map e90 with expected crc 2024-04-02T04:50:34.247 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:33 smithi053 ceph-mon[67180]: failed to encode map e90 with expected crc 2024-04-02T04:50:34.247 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:33 smithi053 ceph-mon[67180]: failed to encode map e90 with expected crc 2024-04-02T04:50:34.327 INFO:teuthology.orchestra.run.smithi053.stdout: 5 207218 10.66 MB/sec execute 344 sec latency 952.344 ms 2024-04-02T04:50:34.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:33 smithi122 ceph-mon[53211]: pgmap v102: 97 pgs: 6 peering, 3 active+undersized+remapped, 88 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 4.2 MiB/s wr, 429 op/s; 1024 KiB/s, 74 keys/s, 52 objects/s recovering 2024-04-02T04:50:34.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:33 smithi122 ceph-mon[53211]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 167/1680 objects degraded (9.940%), 22 pgs degraded) 2024-04-02T04:50:34.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:33 smithi122 ceph-mon[53211]: failed to encode map e90 with expected crc 2024-04-02T04:50:34.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:33 smithi122 ceph-mon[53211]: osdmap e90: 8 total, 8 up, 8 in 2024-04-02T04:50:34.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:33 smithi122 ceph-mon[53211]: failed to encode map e90 with expected crc 2024-04-02T04:50:34.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:33 smithi122 ceph-mon[53211]: failed to encode map e90 with expected crc 2024-04-02T04:50:34.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:33 smithi122 ceph-mon[53211]: failed to encode map e90 with expected crc 2024-04-02T04:50:34.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:33 smithi122 ceph-mon[53211]: failed to encode map e90 with expected crc 2024-04-02T04:50:34.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:33 smithi122 ceph-mon[53211]: failed to encode map e90 with expected crc 2024-04-02T04:50:34.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:33 smithi122 ceph-mon[53211]: failed to encode map e90 with expected crc 2024-04-02T04:50:35.265 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:34 smithi053 ceph-mon[67180]: failed to encode map e91 with expected crc 2024-04-02T04:50:35.266 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:34 smithi053 ceph-mon[67180]: osdmap e91: 8 total, 8 up, 8 in 2024-04-02T04:50:35.266 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:34 smithi053 ceph-mon[67180]: failed to encode map e91 with expected crc 2024-04-02T04:50:35.266 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:34 smithi053 ceph-mon[67180]: failed to encode map e91 with expected crc 2024-04-02T04:50:35.266 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:34 smithi053 ceph-mon[67180]: failed to encode map e91 with expected crc 2024-04-02T04:50:35.266 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:34 smithi053 ceph-mon[67180]: failed to encode map e91 with expected crc 2024-04-02T04:50:35.266 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:34 smithi053 ceph-mon[67180]: failed to encode map e91 with expected crc 2024-04-02T04:50:35.266 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:34 smithi053 ceph-mon[67180]: failed to encode map e91 with expected crc 2024-04-02T04:50:35.327 INFO:teuthology.orchestra.run.smithi053.stdout: 5 207591 10.63 MB/sec execute 345 sec latency 59.726 ms 2024-04-02T04:50:35.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:34 smithi122 ceph-mon[53211]: failed to encode map e91 with expected crc 2024-04-02T04:50:35.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:34 smithi122 ceph-mon[53211]: osdmap e91: 8 total, 8 up, 8 in 2024-04-02T04:50:35.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:34 smithi122 ceph-mon[53211]: failed to encode map e91 with expected crc 2024-04-02T04:50:35.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:34 smithi122 ceph-mon[53211]: failed to encode map e91 with expected crc 2024-04-02T04:50:35.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:34 smithi122 ceph-mon[53211]: failed to encode map e91 with expected crc 2024-04-02T04:50:35.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:34 smithi122 ceph-mon[53211]: failed to encode map e91 with expected crc 2024-04-02T04:50:35.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:34 smithi122 ceph-mon[53211]: failed to encode map e91 with expected crc 2024-04-02T04:50:35.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:34 smithi122 ceph-mon[53211]: failed to encode map e91 with expected crc 2024-04-02T04:50:36.299 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:36 smithi053 ceph-mon[67180]: pgmap v105: 97 pgs: 3 peering, 3 active+undersized+remapped, 91 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 3.9 MiB/s wr, 427 op/s; 1.1 MiB/s, 74 keys/s, 56 objects/s recovering 2024-04-02T04:50:36.299 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:36 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:36.299 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:36 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:36.299 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:36 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:50:36.299 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:36 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:50:36.299 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:36 smithi053 ceph-mon[67180]: pgmap v106: 97 pgs: 3 peering, 3 active+undersized+remapped, 91 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 3.6 MiB/s wr, 398 op/s; 1019 KiB/s, 69 keys/s, 52 objects/s recovering 2024-04-02T04:50:36.300 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:36 smithi053 ceph-mon[67180]: pgmap v107: 97 pgs: 3 peering, 3 active+undersized+remapped, 91 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 4.7 MiB/s wr, 520 op/s; 1.3 MiB/s, 90 keys/s, 69 objects/s recovering 2024-04-02T04:50:36.300 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:36 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:36.300 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:36 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:36.300 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:36 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:50:36.300 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:36 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:50:36.300 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:36 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:50:36.300 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:36 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:50:36.300 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:36 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-02T04:50:36.300 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:36 smithi053 ceph-mon[67180]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-02T04:50:36.300 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:36 smithi053 ceph-mon[67180]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-04-02T04:50:36.328 INFO:teuthology.orchestra.run.smithi053.stdout: 5 207830 10.62 MB/sec execute 346 sec latency 46.185 ms 2024-04-02T04:50:36.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:36 smithi122 ceph-mon[53211]: pgmap v105: 97 pgs: 3 peering, 3 active+undersized+remapped, 91 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 3.9 MiB/s wr, 427 op/s; 1.1 MiB/s, 74 keys/s, 56 objects/s recovering 2024-04-02T04:50:36.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:36 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:36.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:36 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:36.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:36 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:50:36.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:36 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:50:36.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:36 smithi122 ceph-mon[53211]: pgmap v106: 97 pgs: 3 peering, 3 active+undersized+remapped, 91 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 3.6 MiB/s wr, 398 op/s; 1019 KiB/s, 69 keys/s, 52 objects/s recovering 2024-04-02T04:50:36.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:36 smithi122 ceph-mon[53211]: pgmap v107: 97 pgs: 3 peering, 3 active+undersized+remapped, 91 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 4.7 MiB/s wr, 520 op/s; 1.3 MiB/s, 90 keys/s, 69 objects/s recovering 2024-04-02T04:50:36.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:36 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:36.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:36 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:36.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:36 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:50:36.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:36 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:50:36.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:36 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:50:36.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:36 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:50:36.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:36 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-02T04:50:36.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:36 smithi122 ceph-mon[53211]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-02T04:50:36.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:36 smithi122 ceph-mon[53211]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-04-02T04:50:37.328 INFO:teuthology.orchestra.run.smithi053.stdout: 5 208003 10.61 MB/sec execute 347 sec latency 78.990 ms 2024-04-02T04:50:38.328 INFO:teuthology.orchestra.run.smithi053.stdout: 5 208329 10.62 MB/sec execute 348 sec latency 45.634 ms 2024-04-02T04:50:38.372 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:38 smithi053 ceph-mon[67180]: pgmap v108: 97 pgs: 3 peering, 3 active+undersized+remapped, 91 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 315 KiB/s wr, 22 op/s; 65 KiB/s, 4 objects/s recovering 2024-04-02T04:50:38.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:38 smithi122 ceph-mon[53211]: pgmap v108: 97 pgs: 3 peering, 3 active+undersized+remapped, 91 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 315 KiB/s wr, 22 op/s; 65 KiB/s, 4 objects/s recovering 2024-04-02T04:50:39.328 INFO:teuthology.orchestra.run.smithi053.stdout: 5 208590 10.61 MB/sec execute 349 sec latency 83.223 ms 2024-04-02T04:50:40.328 INFO:teuthology.orchestra.run.smithi053.stdout: 5 208895 10.62 MB/sec execute 350 sec latency 54.617 ms 2024-04-02T04:50:40.433 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:40 smithi053 ceph-mon[67180]: pgmap v109: 97 pgs: 97 active+clean; 343 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 3.2 MiB/s wr, 329 op/s; 44 KiB/s, 2 objects/s recovering 2024-04-02T04:50:40.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:40 smithi122 ceph-mon[53211]: pgmap v109: 97 pgs: 97 active+clean; 343 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 3.2 MiB/s wr, 329 op/s; 44 KiB/s, 2 objects/s recovering 2024-04-02T04:50:41.328 INFO:teuthology.orchestra.run.smithi053.stdout: 5 209177 10.61 MB/sec execute 351 sec latency 47.768 ms 2024-04-02T04:50:42.328 INFO:teuthology.orchestra.run.smithi053.stdout: 5 209429 10.59 MB/sec execute 352 sec latency 43.584 ms 2024-04-02T04:50:42.479 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:42 smithi053 ceph-mon[67180]: pgmap v110: 97 pgs: 97 active+clean; 343 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 2.7 MiB/s wr, 284 op/s; 38 KiB/s, 2 objects/s recovering 2024-04-02T04:50:42.479 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:42 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:42.479 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:42 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:50:42.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:42 smithi122 ceph-mon[53211]: pgmap v110: 97 pgs: 97 active+clean; 343 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 2.7 MiB/s wr, 284 op/s; 38 KiB/s, 2 objects/s recovering 2024-04-02T04:50:42.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:42 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:42.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:42 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:50:43.328 INFO:teuthology.orchestra.run.smithi053.stdout: 5 209710 10.57 MB/sec execute 353 sec latency 40.495 ms 2024-04-02T04:50:44.328 INFO:teuthology.orchestra.run.smithi053.stdout: 5 210011 10.56 MB/sec execute 354 sec latency 50.846 ms 2024-04-02T04:50:44.544 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:44 smithi053 ceph-mon[67180]: pgmap v111: 97 pgs: 97 active+clean; 347 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.7 MiB/s wr, 466 op/s 2024-04-02T04:50:44.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:44 smithi122 ceph-mon[53211]: pgmap v111: 97 pgs: 97 active+clean; 347 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.7 MiB/s wr, 466 op/s 2024-04-02T04:50:45.328 INFO:teuthology.orchestra.run.smithi053.stdout: 5 210295 10.55 MB/sec execute 355 sec latency 55.037 ms 2024-04-02T04:50:46.328 INFO:teuthology.orchestra.run.smithi053.stdout: 5 210716 10.56 MB/sec execute 356 sec latency 65.143 ms 2024-04-02T04:50:46.362 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:46 smithi053 ceph-mon[67180]: pgmap v112: 97 pgs: 97 active+clean; 348 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.2 MiB/s wr, 401 op/s 2024-04-02T04:50:46.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:46 smithi122 ceph-mon[53211]: pgmap v112: 97 pgs: 97 active+clean; 348 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.2 MiB/s wr, 401 op/s 2024-04-02T04:50:47.328 INFO:teuthology.orchestra.run.smithi053.stdout: 5 211087 10.54 MB/sec execute 357 sec latency 57.180 ms 2024-04-02T04:50:47.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:47 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:47.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:47 smithi122 ceph-mon[53211]: pgmap v113: 97 pgs: 97 active+clean; 348 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 3.5 MiB/s wr, 334 op/s 2024-04-02T04:50:47.900 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:47 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:47.901 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:47 smithi053 ceph-mon[67180]: pgmap v113: 97 pgs: 97 active+clean; 348 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 3.5 MiB/s wr, 334 op/s 2024-04-02T04:50:48.328 INFO:teuthology.orchestra.run.smithi053.stdout: 5 211385 10.54 MB/sec execute 358 sec latency 56.994 ms 2024-04-02T04:50:49.328 INFO:teuthology.orchestra.run.smithi053.stdout: 5 211675 10.54 MB/sec execute 359 sec latency 43.869 ms 2024-04-02T04:50:50.328 INFO:teuthology.orchestra.run.smithi053.stdout: 5 211984 10.55 MB/sec execute 360 sec latency 49.838 ms 2024-04-02T04:50:50.426 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:50 smithi053 ceph-mon[67180]: pgmap v114: 97 pgs: 97 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.3 MiB/s wr, 538 op/s 2024-04-02T04:50:50.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:50 smithi122 ceph-mon[53211]: pgmap v114: 97 pgs: 97 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.3 MiB/s wr, 538 op/s 2024-04-02T04:50:51.328 INFO:teuthology.orchestra.run.smithi053.stdout: 5 212323 10.55 MB/sec execute 361 sec latency 43.904 ms 2024-04-02T04:50:51.441 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:51 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-02T04:50:51.441 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:51 smithi053 ceph-mon[67180]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-02T04:50:51.441 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:51 smithi053 ceph-mon[67180]: Upgrade: osd.2 is safe to restart 2024-04-02T04:50:51.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:51 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-02T04:50:51.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:51 smithi122 ceph-mon[53211]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-02T04:50:51.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:51 smithi122 ceph-mon[53211]: Upgrade: osd.2 is safe to restart 2024-04-02T04:50:52.328 INFO:teuthology.orchestra.run.smithi053.stdout: 5 212680 10.55 MB/sec execute 362 sec latency 35.971 ms 2024-04-02T04:50:52.463 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:52 smithi053 ceph-mon[67180]: pgmap v115: 97 pgs: 97 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.8 MiB/s wr, 374 op/s 2024-04-02T04:50:52.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:52 smithi122 ceph-mon[53211]: pgmap v115: 97 pgs: 97 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.8 MiB/s wr, 374 op/s 2024-04-02T04:50:53.328 INFO:teuthology.orchestra.run.smithi053.stdout: 5 212926 10.53 MB/sec execute 363 sec latency 69.648 ms 2024-04-02T04:50:54.328 INFO:teuthology.orchestra.run.smithi053.stdout: 5 213146 10.51 MB/sec execute 364 sec latency 70.960 ms 2024-04-02T04:50:54.511 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:54 smithi053 ceph-mon[67180]: pgmap v116: 97 pgs: 97 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.7 MiB/s wr, 553 op/s 2024-04-02T04:50:54.511 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:54 smithi053 ceph-mon[67180]: Upgrade: Updating osd.2 2024-04-02T04:50:54.511 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:54.511 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-02T04:50:54.511 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:50:54.512 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:54 smithi053 ceph-mon[67180]: Deploying daemon osd.2 on smithi053 2024-04-02T04:50:54.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:54 smithi122 ceph-mon[53211]: pgmap v116: 97 pgs: 97 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.7 MiB/s wr, 553 op/s 2024-04-02T04:50:54.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:54 smithi122 ceph-mon[53211]: Upgrade: Updating osd.2 2024-04-02T04:50:54.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:50:54.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-02T04:50:54.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:50:54.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:54 smithi122 ceph-mon[53211]: Deploying daemon osd.2 on smithi053 2024-04-02T04:50:55.328 INFO:teuthology.orchestra.run.smithi053.stdout: 5 213452 10.50 MB/sec execute 365 sec latency 34.943 ms 2024-04-02T04:50:56.328 INFO:teuthology.orchestra.run.smithi053.stdout: 5 213743 10.49 MB/sec execute 366 sec latency 48.418 ms 2024-04-02T04:50:56.566 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:56 smithi053 ceph-mon[67180]: pgmap v117: 97 pgs: 97 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.2 MiB/s wr, 404 op/s 2024-04-02T04:50:56.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:56 smithi122 ceph-mon[53211]: pgmap v117: 97 pgs: 97 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.2 MiB/s wr, 404 op/s 2024-04-02T04:50:57.328 INFO:teuthology.orchestra.run.smithi053.stdout: 5 214130 10.50 MB/sec execute 367 sec latency 38.215 ms 2024-04-02T04:50:57.576 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:57 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:50:57.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:57 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:50:58.329 INFO:teuthology.orchestra.run.smithi053.stdout: 5 214618 10.49 MB/sec execute 368 sec latency 43.245 ms 2024-04-02T04:50:58.582 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:50:58 smithi053 systemd[1]: Stopping Ceph osd.2 for 48e16742-f0aa-11ee-b647-cb9ed24678a4... 2024-04-02T04:50:58.582 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:58 smithi053 ceph-mon[67180]: pgmap v118: 97 pgs: 97 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.0 MiB/s wr, 393 op/s 2024-04-02T04:50:58.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:58 smithi122 ceph-mon[53211]: pgmap v118: 97 pgs: 97 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.0 MiB/s wr, 393 op/s 2024-04-02T04:50:59.088 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:50:58 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2[43072]: 2024-04-02T04:50:58.880+0000 7f0c67592700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T04:50:59.088 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:50:58 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2[43072]: 2024-04-02T04:50:58.880+0000 7f0c67592700 -1 osd.2 91 *** Got signal Terminated *** 2024-04-02T04:50:59.088 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:50:58 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2[43072]: 2024-04-02T04:50:58.880+0000 7f0c67592700 -1 osd.2 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T04:50:59.328 INFO:teuthology.orchestra.run.smithi053.stdout: 5 214971 10.48 MB/sec execute 369 sec latency 72.626 ms 2024-04-02T04:50:59.593 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:50:59 smithi053 ceph-mon[67180]: osd.2 marked itself down and dead 2024-04-02T04:50:59.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:50:59 smithi122 ceph-mon[53211]: osd.2 marked itself down and dead 2024-04-02T04:51:00.328 INFO:teuthology.orchestra.run.smithi053.stdout: 5 215104 10.46 MB/sec execute 370 sec latency 879.736 ms 2024-04-02T04:51:00.438 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:00 smithi053 podman[77724]: 2024-04-02 04:51:00.106206572 +0000 UTC m=+1.355505345 container died 86964c8a94ff47083654f41bcec778abffedf55909fa63bfd801567a85c00da7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2, GIT_BRANCH=HEAD, name=centos-stream, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , architecture=x86_64, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, com.redhat.component=centos-stream-container, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-02T04:51:00.689 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:00 smithi053 ceph-mon[67180]: pgmap v119: 97 pgs: 97 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 5.6 MiB/s wr, 593 op/s 2024-04-02T04:51:00.689 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:00 smithi053 ceph-mon[67180]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-02T04:51:00.689 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:00 smithi053 ceph-mon[67180]: failed to encode map e92 with expected crc 2024-04-02T04:51:00.689 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:00 smithi053 ceph-mon[67180]: osdmap e92: 8 total, 7 up, 8 in 2024-04-02T04:51:00.689 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:00 smithi053 ceph-mon[67180]: failed to encode map e92 with expected crc 2024-04-02T04:51:00.689 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:00 smithi053 ceph-mon[67180]: failed to encode map e92 with expected crc 2024-04-02T04:51:00.689 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:00 smithi053 ceph-mon[67180]: failed to encode map e92 with expected crc 2024-04-02T04:51:00.689 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:00 smithi053 ceph-mon[67180]: failed to encode map e92 with expected crc 2024-04-02T04:51:00.689 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:00 smithi053 ceph-mon[67180]: failed to encode map e92 with expected crc 2024-04-02T04:51:00.689 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:00 smithi053 ceph-mon[67180]: failed to encode map e92 with expected crc 2024-04-02T04:51:00.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:00 smithi122 ceph-mon[53211]: pgmap v119: 97 pgs: 97 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 5.6 MiB/s wr, 593 op/s 2024-04-02T04:51:00.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:00 smithi122 ceph-mon[53211]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-02T04:51:00.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:00 smithi122 ceph-mon[53211]: failed to encode map e92 with expected crc 2024-04-02T04:51:00.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:00 smithi122 ceph-mon[53211]: osdmap e92: 8 total, 7 up, 8 in 2024-04-02T04:51:00.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:00 smithi122 ceph-mon[53211]: failed to encode map e92 with expected crc 2024-04-02T04:51:00.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:00 smithi122 ceph-mon[53211]: failed to encode map e92 with expected crc 2024-04-02T04:51:00.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:00 smithi122 ceph-mon[53211]: failed to encode map e92 with expected crc 2024-04-02T04:51:00.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:00 smithi122 ceph-mon[53211]: failed to encode map e92 with expected crc 2024-04-02T04:51:00.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:00 smithi122 ceph-mon[53211]: failed to encode map e92 with expected crc 2024-04-02T04:51:00.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:00 smithi122 ceph-mon[53211]: failed to encode map e92 with expected crc 2024-04-02T04:51:01.207 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:00 smithi053 podman[77724]: 2024-04-02 04:51:00.988066779 +0000 UTC m=+2.237365557 container cleanup 86964c8a94ff47083654f41bcec778abffedf55909fa63bfd801567a85c00da7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, architecture=x86_64, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, com.redhat.component=centos-stream-container, name=centos-stream, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, io.k8s.display-name=CentOS Stream 8, version=8, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-02T04:51:01.207 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:00 smithi053 bash[77724]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2 2024-04-02T04:51:01.328 INFO:teuthology.orchestra.run.smithi053.stdout: 5 215497 10.48 MB/sec execute 371 sec latency 1022.641 ms 2024-04-02T04:51:01.458 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:01 smithi053 podman[77743]: 2024-04-02 04:51:01.384952366 +0000 UTC m=+1.275323295 container remove 86964c8a94ff47083654f41bcec778abffedf55909fa63bfd801567a85c00da7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, GIT_CLEAN=True, vcs-type=git, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, ceph=True, RELEASE=HEAD, com.redhat.component=centos-stream-container, release=754, version=8, GIT_BRANCH=HEAD) 2024-04-02T04:51:01.709 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:01 smithi053 ceph-mon[67180]: failed to encode map e93 with expected crc 2024-04-02T04:51:01.709 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:01 smithi053 ceph-mon[67180]: osdmap e93: 8 total, 7 up, 8 in 2024-04-02T04:51:01.709 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:01 smithi053 ceph-mon[67180]: failed to encode map e93 with expected crc 2024-04-02T04:51:01.709 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:01 smithi053 ceph-mon[67180]: failed to encode map e93 with expected crc 2024-04-02T04:51:01.709 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:01 smithi053 ceph-mon[67180]: failed to encode map e93 with expected crc 2024-04-02T04:51:01.709 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:01 smithi053 ceph-mon[67180]: failed to encode map e93 with expected crc 2024-04-02T04:51:01.709 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:01 smithi053 ceph-mon[67180]: failed to encode map e93 with expected crc 2024-04-02T04:51:01.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:01 smithi122 ceph-mon[53211]: failed to encode map e93 with expected crc 2024-04-02T04:51:01.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:01 smithi122 ceph-mon[53211]: osdmap e93: 8 total, 7 up, 8 in 2024-04-02T04:51:01.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:01 smithi122 ceph-mon[53211]: failed to encode map e93 with expected crc 2024-04-02T04:51:01.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:01 smithi122 ceph-mon[53211]: failed to encode map e93 with expected crc 2024-04-02T04:51:01.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:01 smithi122 ceph-mon[53211]: failed to encode map e93 with expected crc 2024-04-02T04:51:01.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:01 smithi122 ceph-mon[53211]: failed to encode map e93 with expected crc 2024-04-02T04:51:01.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:01 smithi122 ceph-mon[53211]: failed to encode map e93 with expected crc 2024-04-02T04:51:01.963 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:01 smithi053 podman[77825]: 2024-04-02 04:51:01.712173545 +0000 UTC m=+0.026308994 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:51:01.963 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:01 smithi053 podman[77825]: 2024-04-02 04:51:01.881147662 +0000 UTC m=+0.195283084 container create 2720e58248a7576c08ebc47d70f148bbfb409cdf9d4f099e71fd39039836a1ef (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, RELEASE=squid-6cec793, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-02T04:51:02.328 INFO:teuthology.orchestra.run.smithi053.stdout: 5 215893 10.49 MB/sec execute 372 sec latency 57.166 ms 2024-04-02T04:51:02.468 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:02 smithi053 podman[77825]: 2024-04-02 04:51:02.254249083 +0000 UTC m=+0.568384516 container init 2720e58248a7576c08ebc47d70f148bbfb409cdf9d4f099e71fd39039836a1ef (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-6cec793, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=) 2024-04-02T04:51:02.468 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:02 smithi053 podman[77825]: 2024-04-02 04:51:02.259959373 +0000 UTC m=+0.574094793 container start 2720e58248a7576c08ebc47d70f148bbfb409cdf9d4f099e71fd39039836a1ef (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-6cec793, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326) 2024-04-02T04:51:02.468 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:02 smithi053 ceph-mon[67180]: pgmap v122: 97 pgs: 7 stale+active+clean, 90 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 2.8 MiB/s wr, 315 op/s 2024-04-02T04:51:02.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:02 smithi122 ceph-mon[53211]: pgmap v122: 97 pgs: 7 stale+active+clean, 90 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 2.8 MiB/s wr, 315 op/s 2024-04-02T04:51:03.222 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:02 smithi053 podman[77825]: 2024-04-02 04:51:02.969105349 +0000 UTC m=+1.283240782 container attach 2720e58248a7576c08ebc47d70f148bbfb409cdf9d4f099e71fd39039836a1ef (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-6cec793, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-02T04:51:03.222 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:02 smithi053 podman[77825]: 2024-04-02 04:51:02.970059567 +0000 UTC m=+1.284194995 container died 2720e58248a7576c08ebc47d70f148bbfb409cdf9d4f099e71fd39039836a1ef (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-6cec793, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T04:51:03.328 INFO:teuthology.orchestra.run.smithi053.stdout: 5 216294 10.49 MB/sec execute 373 sec latency 44.501 ms 2024-04-02T04:51:03.388 INFO:teuthology.orchestra.run.smithi053.stdout:true 2024-04-02T04:51:03.741 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:03 smithi053 ceph-mon[67180]: Health check failed: Degraded data redundancy: 143/1653 objects degraded (8.651%), 19 pgs degraded (PG_DEGRADED) 2024-04-02T04:51:03.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:03 smithi122 ceph-mon[53211]: Health check failed: Degraded data redundancy: 143/1653 objects degraded (8.651%), 19 pgs degraded (PG_DEGRADED) 2024-04-02T04:51:04.256 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:04 smithi053 podman[77851]: 2024-04-02 04:51:04.187511645 +0000 UTC m=+1.447447416 container remove 2720e58248a7576c08ebc47d70f148bbfb409cdf9d4f099e71fd39039836a1ef (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, RELEASE=squid-6cec793, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux ) 2024-04-02T04:51:04.256 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:04 smithi053 systemd[1]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.2.service: Deactivated successfully. 2024-04-02T04:51:04.256 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:04 smithi053 systemd[1]: Stopped Ceph osd.2 for 48e16742-f0aa-11ee-b647-cb9ed24678a4. 2024-04-02T04:51:04.256 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:04 smithi053 systemd[1]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.2.service: Consumed 3min 3.547s CPU time. 2024-04-02T04:51:04.329 INFO:teuthology.orchestra.run.smithi053.stdout: 5 216541 10.48 MB/sec execute 374 sec latency 42.553 ms 2024-04-02T04:51:04.439 INFO:teuthology.orchestra.run.smithi053.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-02T04:51:04.439 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager.smithi053 smithi053 *:9093,9094 running (5m) 35s ago 14m 24.3M - 0.23.0 ba2b418f427c bf50c1cd2b3d 2024-04-02T04:51:04.439 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi053 smithi053 running (115s) 35s ago 14m 6660k - 19.0.0-1979-g6cec793d 78f830190839 4ce8915e0157 2024-04-02T04:51:04.439 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi122 smithi122 running (109s) 105s ago 14m 6660k - 19.0.0-1979-g6cec793d 78f830190839 e2b5369e6fdf 2024-04-02T04:51:04.440 INFO:teuthology.orchestra.run.smithi053.stdout:grafana.smithi053 smithi053 *:3000 error 35s ago 14m - - 2024-04-02T04:51:04.440 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi053.nfrfot smithi053 running (8m) 35s ago 8m 37.3M - 17.2.0 e1d6a67b021e c94e9e643514 2024-04-02T04:51:04.440 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi122.mwoyoh smithi122 running (8m) 105s ago 8m 15.6M - 17.2.0 e1d6a67b021e 73376d9ee4b8 2024-04-02T04:51:04.440 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi053.legdmv smithi053 *:8443,9283,8765 running (4m) 35s ago 15m 503M - 19.0.0-1979-g6cec793d 78f830190839 c12d516a31bc 2024-04-02T04:51:04.440 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi122.cpwlyk smithi122 *:8443,9283,8765 running (3m) 105s ago 13m 449M - 19.0.0-1979-g6cec793d 78f830190839 9a545c5e7d81 2024-04-02T04:51:04.440 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi053 smithi053 running (3m) 35s ago 16m 49.4M 2048M 19.0.0-1979-g6cec793d 78f830190839 f53a44f19c00 2024-04-02T04:51:04.440 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi122 smithi122 running (2m) 105s ago 13m 34.8M 2048M 19.0.0-1979-g6cec793d 78f830190839 44b200fed3c5 2024-04-02T04:51:04.440 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.0.0.smithi053.mhkdcb smithi053 *:2049 running (8m) 35s ago 8m 287M - 4.0 e1d6a67b021e 07e5da7bd2c5 2024-04-02T04:51:04.440 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.1.0.smithi122.mauvsb smithi122 *:2049 running (8m) 105s ago 8m 81.7M - 4.0 e1d6a67b021e 4f866685005e 2024-04-02T04:51:04.440 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi053 smithi053 *:9100 running (5m) 35s ago 14m 18.9M - 1.3.1 1dbe0e931976 773b71971e67 2024-04-02T04:51:04.440 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi122 smithi122 *:9100 running (4m) 105s ago 13m 15.7M - 1.3.1 1dbe0e931976 475d439c7980 2024-04-02T04:51:04.440 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 smithi053 running (78s) 35s ago 13m 235M 1937M 19.0.0-1979-g6cec793d 78f830190839 490c346c7639 2024-04-02T04:51:04.440 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 smithi053 running (44s) 35s ago 12m 12.1M 1937M 19.0.0-1979-g6cec793d 78f830190839 e70e40949c2f 2024-04-02T04:51:04.440 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 smithi053 running (12m) 35s ago 12m 537M 1937M 17.2.0 e1d6a67b021e 86964c8a94ff 2024-04-02T04:51:04.440 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 smithi053 running (11m) 35s ago 11m 367M 1937M 17.2.0 e1d6a67b021e 7e8b9e98ae2b 2024-04-02T04:51:04.441 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 smithi122 running (11m) 105s ago 11m 557M 2704M 17.2.0 e1d6a67b021e 46076afcd7bc 2024-04-02T04:51:04.441 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 smithi122 running (11m) 105s ago 11m 389M 2704M 17.2.0 e1d6a67b021e e687a2f6da4d 2024-04-02T04:51:04.441 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 smithi122 running (10m) 105s ago 10m 568M 2704M 17.2.0 e1d6a67b021e 9e371df9a839 2024-04-02T04:51:04.441 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 smithi122 running (10m) 105s ago 10m 480M 2704M 17.2.0 e1d6a67b021e 9b3cc398254d 2024-04-02T04:51:04.441 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus.smithi053 smithi053 *:9095 running (3m) 35s ago 13m 67.4M - 2.33.4 514e6a882f6e 1dd55399da10 2024-04-02T04:51:04.770 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:04 smithi053 systemd[1]: Starting Ceph osd.2 for 48e16742-f0aa-11ee-b647-cb9ed24678a4... 2024-04-02T04:51:04.770 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:04 smithi053 ceph-mon[67180]: pgmap v123: 97 pgs: 5 active+undersized, 19 active+undersized+degraded, 73 active+clean; 344 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.6 MiB/s wr, 579 op/s; 143/1653 objects degraded (8.651%) 2024-04-02T04:51:04.770 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:04 smithi053 ceph-mon[67180]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:51:04.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:04 smithi122 ceph-mon[53211]: pgmap v123: 97 pgs: 5 active+undersized, 19 active+undersized+degraded, 73 active+clean; 344 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.6 MiB/s wr, 579 op/s; 143/1653 objects degraded (8.651%) 2024-04-02T04:51:04.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:04 smithi122 ceph-mon[53211]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:51:05.013 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:51:05.013 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": { 2024-04-02T04:51:05.013 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:51:05.013 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:51:05.013 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": { 2024-04-02T04:51:05.013 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:51:05.013 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:51:05.013 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": { 2024-04-02T04:51:05.013 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-04-02T04:51:05.013 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:51:05.013 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:51:05.014 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": { 2024-04-02T04:51:05.014 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:51:05.014 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:51:05.014 INFO:teuthology.orchestra.run.smithi053.stdout: "overall": { 2024-04-02T04:51:05.014 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-04-02T04:51:05.014 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 6 2024-04-02T04:51:05.014 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-02T04:51:05.014 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:51:05.024 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:04 smithi053 podman[78045]: 2024-04-02 04:51:04.715239641 +0000 UTC m=+0.024241287 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:51:05.275 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:05 smithi053 podman[78045]: 2024-04-02 04:51:05.039226469 +0000 UTC m=+0.348228082 container create 1a1e2ef3c8e3601aefac65eb50ae1793ead71242bdd910a4efd9a2521539864e (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2-activate, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-6cec793, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-02T04:51:05.329 INFO:teuthology.orchestra.run.smithi053.stdout: 5 216909 10.47 MB/sec execute 375 sec latency 38.578 ms 2024-04-02T04:51:05.489 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:51:05.489 INFO:teuthology.orchestra.run.smithi053.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae", 2024-04-02T04:51:05.489 INFO:teuthology.orchestra.run.smithi053.stdout: "in_progress": true, 2024-04-02T04:51:05.489 INFO:teuthology.orchestra.run.smithi053.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-02T04:51:05.489 INFO:teuthology.orchestra.run.smithi053.stdout: "services_complete": [ 2024-04-02T04:51:05.489 INFO:teuthology.orchestra.run.smithi053.stdout: "mon", 2024-04-02T04:51:05.489 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr", 2024-04-02T04:51:05.489 INFO:teuthology.orchestra.run.smithi053.stdout: "crash" 2024-04-02T04:51:05.489 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-04-02T04:51:05.490 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": "8/23 daemons upgraded", 2024-04-02T04:51:05.490 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "Currently upgrading osd daemons", 2024-04-02T04:51:05.490 INFO:teuthology.orchestra.run.smithi053.stdout: "is_paused": false 2024-04-02T04:51:05.490 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:51:05.528 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:05 smithi053 podman[78045]: 2024-04-02 04:51:05.404698884 +0000 UTC m=+0.713700502 container init 1a1e2ef3c8e3601aefac65eb50ae1793ead71242bdd910a4efd9a2521539864e (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2-activate, ceph=True, RELEASE=squid-6cec793, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-02T04:51:05.528 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:05 smithi053 podman[78045]: 2024-04-02 04:51:05.409146935 +0000 UTC m=+0.718148548 container start 1a1e2ef3c8e3601aefac65eb50ae1793ead71242bdd910a4efd9a2521539864e (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2-activate, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-6cec793, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-02T04:51:05.787 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:05 smithi053 podman[78045]: 2024-04-02 04:51:05.588075927 +0000 UTC m=+0.897077541 container attach 1a1e2ef3c8e3601aefac65eb50ae1793ead71242bdd910a4efd9a2521539864e (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2-activate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-6cec793, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5) 2024-04-02T04:51:05.787 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:05 smithi053 ceph-mon[67180]: from='client.34258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:51:05.787 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:05 smithi053 ceph-mon[67180]: from='client.34262 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:51:05.787 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:05 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/244275604' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:51:05.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:05 smithi122 ceph-mon[53211]: from='client.34258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:51:05.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:05 smithi122 ceph-mon[53211]: from='client.34262 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:51:05.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:05 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/244275604' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:51:06.070 INFO:teuthology.orchestra.run.smithi053.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 156/1650 objects degraded (9.455%), 20 pgs degraded 2024-04-02T04:51:06.070 INFO:teuthology.orchestra.run.smithi053.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-02T04:51:06.070 INFO:teuthology.orchestra.run.smithi053.stdout: daemon grafana.smithi053 on smithi053 is in error state 2024-04-02T04:51:06.070 INFO:teuthology.orchestra.run.smithi053.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-02T04:51:06.070 INFO:teuthology.orchestra.run.smithi053.stdout: osd.2 (root=default,host=smithi053) is down 2024-04-02T04:51:06.070 INFO:teuthology.orchestra.run.smithi053.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 156/1650 objects degraded (9.455%), 20 pgs degraded 2024-04-02T04:51:06.070 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.1 is active+undersized+degraded, acting [3,0] 2024-04-02T04:51:06.070 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.3 is active+undersized+degraded, acting [5,7] 2024-04-02T04:51:06.070 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.7 is active+undersized+degraded, acting [6,7] 2024-04-02T04:51:06.071 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.b is active+undersized+degraded, acting [4,5] 2024-04-02T04:51:06.071 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.c is active+undersized+degraded, acting [5,0] 2024-04-02T04:51:06.071 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.e is active+undersized+degraded, acting [3,7] 2024-04-02T04:51:06.071 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.10 is active+undersized+degraded, acting [1,0] 2024-04-02T04:51:06.071 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.16 is active+undersized+degraded, acting [5,6] 2024-04-02T04:51:06.071 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.17 is active+undersized+degraded, acting [6,5] 2024-04-02T04:51:06.071 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.1c is active+undersized+degraded, acting [7,5] 2024-04-02T04:51:06.071 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.0 is active+undersized+degraded, acting [1,6] 2024-04-02T04:51:06.071 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.4 is active+undersized+degraded, acting [1,5] 2024-04-02T04:51:06.071 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.5 is active+undersized+degraded, acting [5,3] 2024-04-02T04:51:06.071 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.9 is active+undersized+degraded, acting [4,7] 2024-04-02T04:51:06.071 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.13 is active+undersized+degraded, acting [7,4] 2024-04-02T04:51:06.071 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.1a is active+undersized+degraded, acting [4,1] 2024-04-02T04:51:06.071 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.1e is active+undersized+degraded, acting [3,6] 2024-04-02T04:51:06.071 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.1f is active+undersized+degraded, acting [0,5] 2024-04-02T04:51:06.071 INFO:teuthology.orchestra.run.smithi053.stdout: pg 4.a is active+undersized+degraded, acting [6,7] 2024-04-02T04:51:06.072 INFO:teuthology.orchestra.run.smithi053.stdout: pg 4.19 is active+undersized+degraded, acting [3,0] 2024-04-02T04:51:06.294 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:06 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2-activate[78082]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-02T04:51:06.294 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:06 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2-activate[78082]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-02T04:51:06.294 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:06 smithi053 bash[78045]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-02T04:51:06.294 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:06 smithi053 bash[78045]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-02T04:51:06.294 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:06 smithi053 bash[78045]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-02T04:51:06.294 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:06 smithi053 bash[78045]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-02T04:51:06.294 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:06 smithi053 bash[78045]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-02T04:51:06.294 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:06 smithi053 bash[78045]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-02T04:51:06.294 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:06 smithi053 bash[78045]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-02T04:51:06.295 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:06 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2-activate[78082]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-02T04:51:06.295 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:06 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2-activate[78082]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-02T04:51:06.295 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:06 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2-activate[78082]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-02T04:51:06.295 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:06 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2-activate[78082]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-02T04:51:06.295 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:06 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2-activate[78082]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-02T04:51:06.295 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:06 smithi053 podman[78045]: 2024-04-02 04:51:06.108054357 +0000 UTC m=+1.417055980 container died 1a1e2ef3c8e3601aefac65eb50ae1793ead71242bdd910a4efd9a2521539864e (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, GIT_CLEAN=True, RELEASE=squid-6cec793, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-02T04:51:06.329 INFO:teuthology.orchestra.run.smithi053.stdout: 5 217265 10.47 MB/sec execute 376 sec latency 47.234 ms 2024-04-02T04:51:06.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:06 smithi053 ceph-mon[67180]: pgmap v124: 97 pgs: 6 active+undersized, 20 active+undersized+degraded, 71 active+clean; 344 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 6.0 MiB/s wr, 594 op/s; 156/1650 objects degraded (9.455%) 2024-04-02T04:51:06.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:06 smithi053 ceph-mon[67180]: from='client.34270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:51:06.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:06 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/2964199630' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:51:06.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:06 smithi122 ceph-mon[53211]: pgmap v124: 97 pgs: 6 active+undersized, 20 active+undersized+degraded, 71 active+clean; 344 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 6.0 MiB/s wr, 594 op/s; 156/1650 objects degraded (9.455%) 2024-04-02T04:51:06.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:06 smithi122 ceph-mon[53211]: from='client.34270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:51:06.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:06 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/2964199630' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:51:07.330 INFO:teuthology.orchestra.run.smithi053.stdout: 5 217696 10.47 MB/sec execute 377 sec latency 47.957 ms 2024-04-02T04:51:07.583 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:07 smithi053 ceph-mon[67180]: pgmap v125: 97 pgs: 6 active+undersized, 20 active+undersized+degraded, 71 active+clean; 344 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 3.5 MiB/s wr, 294 op/s; 156/1650 objects degraded (9.455%) 2024-04-02T04:51:07.583 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:07 smithi053 podman[78229]: 2024-04-02 04:51:07.494444596 +0000 UTC m=+1.377379193 container remove 1a1e2ef3c8e3601aefac65eb50ae1793ead71242bdd910a4efd9a2521539864e (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, RELEASE=squid-6cec793) 2024-04-02T04:51:07.834 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:07 smithi053 podman[78290]: 2024-04-02 04:51:07.653503751 +0000 UTC m=+0.024223517 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:51:07.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:07 smithi122 ceph-mon[53211]: pgmap v125: 97 pgs: 6 active+undersized, 20 active+undersized+degraded, 71 active+clean; 344 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 3.5 MiB/s wr, 294 op/s; 156/1650 objects degraded (9.455%) 2024-04-02T04:51:08.001 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:07 smithi053 podman[78290]: 2024-04-02 04:51:07.85036133 +0000 UTC m=+0.221081079 container create 4f053827e0dd38ab53819701163449278eb07e84ece8c26b1ce8292bd7b7f98f (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2, io.buildah.version=1.33.5, RELEASE=squid-6cec793, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True) 2024-04-02T04:51:08.330 INFO:teuthology.orchestra.run.smithi053.stdout: 5 218316 10.47 MB/sec execute 378 sec latency 31.875 ms 2024-04-02T04:51:08.345 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:08 smithi053 podman[78290]: 2024-04-02 04:51:08.237954593 +0000 UTC m=+0.608674355 container init 4f053827e0dd38ab53819701163449278eb07e84ece8c26b1ce8292bd7b7f98f (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-6cec793, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-02T04:51:08.345 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:08 smithi053 podman[78290]: 2024-04-02 04:51:08.245046658 +0000 UTC m=+0.615766404 container start 4f053827e0dd38ab53819701163449278eb07e84ece8c26b1ce8292bd7b7f98f (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-6cec793, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-02T04:51:08.345 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:08 smithi053 ceph-osd[78308]: -- 172.21.15.53:0/3631533417 <== mon.0 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556288d9b680 con 0x556288d65000 2024-04-02T04:51:08.596 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:08 smithi053 bash[78290]: 4f053827e0dd38ab53819701163449278eb07e84ece8c26b1ce8292bd7b7f98f 2024-04-02T04:51:08.596 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:08 smithi053 systemd[1]: Started Ceph osd.2 for 48e16742-f0aa-11ee-b647-cb9ed24678a4. 2024-04-02T04:51:08.847 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:08 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2[78304]: 2024-04-02T04:51:08.826+0000 7fc308d437c0 -1 Falling back to public interface 2024-04-02T04:51:09.330 INFO:teuthology.orchestra.run.smithi053.stdout: 5 218834 10.48 MB/sec execute 379 sec latency 47.640 ms 2024-04-02T04:51:09.614 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:09 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:09.614 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:09 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:09.614 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:09 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:51:09.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:09 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:09.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:09 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:09.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:09 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:51:10.330 INFO:teuthology.orchestra.run.smithi053.stdout: 5 219222 10.50 MB/sec execute 380 sec latency 55.056 ms 2024-04-02T04:51:10.632 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:10 smithi053 ceph-mon[67180]: pgmap v126: 97 pgs: 6 active+undersized, 20 active+undersized+degraded, 71 active+clean; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.5 MiB/s wr, 532 op/s; 164/1602 objects degraded (10.237%) 2024-04-02T04:51:10.632 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:10 smithi053 ceph-mon[67180]: Health check update: Degraded data redundancy: 164/1602 objects degraded (10.237%), 20 pgs degraded (PG_DEGRADED) 2024-04-02T04:51:10.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:10 smithi122 ceph-mon[53211]: pgmap v126: 97 pgs: 6 active+undersized, 20 active+undersized+degraded, 71 active+clean; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.5 MiB/s wr, 532 op/s; 164/1602 objects degraded (10.237%) 2024-04-02T04:51:10.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:10 smithi122 ceph-mon[53211]: Health check update: Degraded data redundancy: 164/1602 objects degraded (10.237%), 20 pgs degraded (PG_DEGRADED) 2024-04-02T04:51:11.330 INFO:teuthology.orchestra.run.smithi053.stdout: 5 219561 10.50 MB/sec execute 381 sec latency 49.311 ms 2024-04-02T04:51:12.330 INFO:teuthology.orchestra.run.smithi053.stdout: 5 219857 10.50 MB/sec execute 382 sec latency 54.167 ms 2024-04-02T04:51:12.662 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:12 smithi053 ceph-mon[67180]: pgmap v127: 97 pgs: 6 active+undersized, 20 active+undersized+degraded, 71 active+clean; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 5.0 MiB/s wr, 483 op/s; 164/1602 objects degraded (10.237%) 2024-04-02T04:51:12.662 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:12 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:12.662 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:12 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:51:12.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:12 smithi122 ceph-mon[53211]: pgmap v127: 97 pgs: 6 active+undersized, 20 active+undersized+degraded, 71 active+clean; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 5.0 MiB/s wr, 483 op/s; 164/1602 objects degraded (10.237%) 2024-04-02T04:51:12.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:12 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:12.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:12 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:51:13.330 INFO:teuthology.orchestra.run.smithi053.stdout: 5 220048 10.48 MB/sec execute 383 sec latency 47.981 ms 2024-04-02T04:51:14.202 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:14 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2[78304]: 2024-04-02T04:51:14.141+0000 7fc308d437c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-04-02T04:51:14.330 INFO:teuthology.orchestra.run.smithi053.stdout: 5 220210 10.46 MB/sec execute 384 sec latency 63.867 ms 2024-04-02T04:51:14.705 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:14 smithi053 ceph-mon[67180]: pgmap v128: 97 pgs: 6 active+undersized, 20 active+undersized+degraded, 71 active+clean; 343 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.8 MiB/s wr, 664 op/s; 170/1668 objects degraded (10.192%) 2024-04-02T04:51:14.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:14 smithi122 ceph-mon[53211]: pgmap v128: 97 pgs: 6 active+undersized, 20 active+undersized+degraded, 71 active+clean; 343 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.8 MiB/s wr, 664 op/s; 170/1668 objects degraded (10.192%) 2024-04-02T04:51:15.330 INFO:teuthology.orchestra.run.smithi053.stdout: 5 220476 10.44 MB/sec execute 385 sec latency 60.777 ms 2024-04-02T04:51:15.484 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:15 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2[78304]: 2024-04-02T04:51:15.329+0000 7fc308d437c0 -1 osd.2 91 log_to_monitors true 2024-04-02T04:51:15.766 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:15 smithi053 ceph-mon[67180]: from='osd.2 [v2:172.21.15.53:6818/4043555433,v1:172.21.15.53:6819/4043555433]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-02T04:51:15.766 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:15 smithi053 ceph-mon[67180]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-02T04:51:15.766 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:51:15 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2[78304]: 2024-04-02T04:51:15.556+0000 7fc2fa19b700 -1 osd.2 91 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T04:51:15.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:15 smithi122 ceph-mon[53211]: from='osd.2 [v2:172.21.15.53:6818/4043555433,v1:172.21.15.53:6819/4043555433]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-02T04:51:15.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:15 smithi122 ceph-mon[53211]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-02T04:51:16.331 INFO:teuthology.orchestra.run.smithi053.stdout: 5 220706 10.43 MB/sec execute 386 sec latency 63.232 ms 2024-04-02T04:51:16.796 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:16 smithi122 ceph-mon[53211]: pgmap v129: 97 pgs: 6 active+undersized, 20 active+undersized+degraded, 71 active+clean; 342 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.9 MiB/s wr, 485 op/s; 170/1671 objects degraded (10.174%) 2024-04-02T04:51:16.797 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:16 smithi122 ceph-mon[53211]: Health check update: Degraded data redundancy: 170/1671 objects degraded (10.174%), 20 pgs degraded (PG_DEGRADED) 2024-04-02T04:51:16.797 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:16 smithi122 ceph-mon[53211]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-02T04:51:16.797 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:16 smithi122 ceph-mon[53211]: failed to encode map e94 with expected crc 2024-04-02T04:51:16.797 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:16 smithi122 ceph-mon[53211]: osdmap e94: 8 total, 7 up, 8 in 2024-04-02T04:51:16.797 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:16 smithi122 ceph-mon[53211]: failed to encode map e94 with expected crc 2024-04-02T04:51:16.797 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:16 smithi122 ceph-mon[53211]: from='osd.2 [v2:172.21.15.53:6818/4043555433,v1:172.21.15.53:6819/4043555433]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-02T04:51:16.797 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:16 smithi122 ceph-mon[53211]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-02T04:51:16.797 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:16 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:16.797 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:16 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:16.801 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:16 smithi053 ceph-mon[67180]: pgmap v129: 97 pgs: 6 active+undersized, 20 active+undersized+degraded, 71 active+clean; 342 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.9 MiB/s wr, 485 op/s; 170/1671 objects degraded (10.174%) 2024-04-02T04:51:16.801 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:16 smithi053 ceph-mon[67180]: Health check update: Degraded data redundancy: 170/1671 objects degraded (10.174%), 20 pgs degraded (PG_DEGRADED) 2024-04-02T04:51:16.801 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:16 smithi053 ceph-mon[67180]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-02T04:51:16.801 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:16 smithi053 ceph-mon[67180]: failed to encode map e94 with expected crc 2024-04-02T04:51:16.801 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:16 smithi053 ceph-mon[67180]: osdmap e94: 8 total, 7 up, 8 in 2024-04-02T04:51:16.801 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:16 smithi053 ceph-mon[67180]: failed to encode map e94 with expected crc 2024-04-02T04:51:16.801 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:16 smithi053 ceph-mon[67180]: from='osd.2 [v2:172.21.15.53:6818/4043555433,v1:172.21.15.53:6819/4043555433]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-02T04:51:16.801 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:16 smithi053 ceph-mon[67180]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-02T04:51:16.801 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:16 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:16.801 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:16 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:17.331 INFO:teuthology.orchestra.run.smithi053.stdout: 5 220823 10.42 MB/sec execute 387 sec latency 731.823 ms 2024-04-02T04:51:17.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:17 smithi122 ceph-mon[53211]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-02T04:51:17.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:17 smithi122 ceph-mon[53211]: failed to encode map e95 with expected crc 2024-04-02T04:51:17.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:17 smithi122 ceph-mon[53211]: osd.2 [v2:172.21.15.53:6818/4043555433,v1:172.21.15.53:6819/4043555433] boot 2024-04-02T04:51:17.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:17 smithi122 ceph-mon[53211]: osdmap e95: 8 total, 8 up, 8 in 2024-04-02T04:51:17.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:17 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:51:17.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:17 smithi122 ceph-mon[53211]: osdmap e96: 8 total, 8 up, 8 in 2024-04-02T04:51:18.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:17 smithi053 ceph-mon[67180]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-02T04:51:18.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:17 smithi053 ceph-mon[67180]: failed to encode map e95 with expected crc 2024-04-02T04:51:18.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:17 smithi053 ceph-mon[67180]: osd.2 [v2:172.21.15.53:6818/4043555433,v1:172.21.15.53:6819/4043555433] boot 2024-04-02T04:51:18.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:17 smithi053 ceph-mon[67180]: osdmap e95: 8 total, 8 up, 8 in 2024-04-02T04:51:18.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:17 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T04:51:18.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:17 smithi053 ceph-mon[67180]: osdmap e96: 8 total, 8 up, 8 in 2024-04-02T04:51:18.331 INFO:teuthology.orchestra.run.smithi053.stdout: 5 220833 10.39 MB/sec execute 388 sec latency 1731.850 ms 2024-04-02T04:51:18.744 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:18 smithi053 ceph-mon[67180]: pgmap v132: 97 pgs: 6 active+undersized, 20 active+undersized+degraded, 71 active+clean; 342 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.8 MiB/s wr, 352 op/s; 170/1671 objects degraded (10.174%) 2024-04-02T04:51:18.744 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:18 smithi053 ceph-mon[67180]: failed to encode map e96 with expected crc 2024-04-02T04:51:18.744 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:18 smithi053 ceph-mon[67180]: failed to encode map e96 with expected crc 2024-04-02T04:51:18.744 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:18 smithi053 ceph-mon[67180]: failed to encode map e96 with expected crc 2024-04-02T04:51:18.744 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:18 smithi053 ceph-mon[67180]: failed to encode map e96 with expected crc 2024-04-02T04:51:18.744 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:18 smithi053 ceph-mon[67180]: failed to encode map e96 with expected crc 2024-04-02T04:51:18.744 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:18 smithi053 ceph-mon[67180]: failed to encode map e96 with expected crc 2024-04-02T04:51:18.744 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:18 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:18.744 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:18 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:18.744 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:18 smithi053 ceph-mon[67180]: osdmap e97: 8 total, 8 up, 8 in 2024-04-02T04:51:18.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:18 smithi122 ceph-mon[53211]: pgmap v132: 97 pgs: 6 active+undersized, 20 active+undersized+degraded, 71 active+clean; 342 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.8 MiB/s wr, 352 op/s; 170/1671 objects degraded (10.174%) 2024-04-02T04:51:18.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:18 smithi122 ceph-mon[53211]: failed to encode map e96 with expected crc 2024-04-02T04:51:18.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:18 smithi122 ceph-mon[53211]: failed to encode map e96 with expected crc 2024-04-02T04:51:18.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:18 smithi122 ceph-mon[53211]: failed to encode map e96 with expected crc 2024-04-02T04:51:18.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:18 smithi122 ceph-mon[53211]: failed to encode map e96 with expected crc 2024-04-02T04:51:18.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:18 smithi122 ceph-mon[53211]: failed to encode map e96 with expected crc 2024-04-02T04:51:18.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:18 smithi122 ceph-mon[53211]: failed to encode map e96 with expected crc 2024-04-02T04:51:18.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:18 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:18.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:18 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:18.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:18 smithi122 ceph-mon[53211]: osdmap e97: 8 total, 8 up, 8 in 2024-04-02T04:51:19.331 INFO:teuthology.orchestra.run.smithi053.stdout: 5 221008 10.38 MB/sec execute 389 sec latency 2214.044 ms 2024-04-02T04:51:19.754 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:19 smithi053 ceph-mon[67180]: failed to encode map e97 with expected crc 2024-04-02T04:51:19.754 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:19 smithi053 ceph-mon[67180]: failed to encode map e97 with expected crc 2024-04-02T04:51:19.754 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:19 smithi053 ceph-mon[67180]: failed to encode map e97 with expected crc 2024-04-02T04:51:19.754 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:19 smithi053 ceph-mon[67180]: failed to encode map e97 with expected crc 2024-04-02T04:51:19.754 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:19 smithi053 ceph-mon[67180]: failed to encode map e97 with expected crc 2024-04-02T04:51:19.754 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:19 smithi053 ceph-mon[67180]: failed to encode map e97 with expected crc 2024-04-02T04:51:19.754 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:19 smithi053 ceph-mon[67180]: failed to encode map e97 with expected crc 2024-04-02T04:51:19.754 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:19 smithi053 ceph-mon[67180]: pgmap v135: 97 pgs: 3 peering, 3 activating, 6 remapped+peering, 2 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+undersized+degraded, 77 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 4.0 MiB/s wr, 390 op/s; 52/1560 objects degraded (3.333%); 39 KiB/s, 4 objects/s recovering 2024-04-02T04:51:19.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:19 smithi122 ceph-mon[53211]: failed to encode map e97 with expected crc 2024-04-02T04:51:19.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:19 smithi122 ceph-mon[53211]: failed to encode map e97 with expected crc 2024-04-02T04:51:19.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:19 smithi122 ceph-mon[53211]: failed to encode map e97 with expected crc 2024-04-02T04:51:19.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:19 smithi122 ceph-mon[53211]: failed to encode map e97 with expected crc 2024-04-02T04:51:19.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:19 smithi122 ceph-mon[53211]: failed to encode map e97 with expected crc 2024-04-02T04:51:19.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:19 smithi122 ceph-mon[53211]: failed to encode map e97 with expected crc 2024-04-02T04:51:19.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:19 smithi122 ceph-mon[53211]: failed to encode map e97 with expected crc 2024-04-02T04:51:19.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:19 smithi122 ceph-mon[53211]: pgmap v135: 97 pgs: 3 peering, 3 activating, 6 remapped+peering, 2 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+undersized+degraded, 77 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 4.0 MiB/s wr, 390 op/s; 52/1560 objects degraded (3.333%); 39 KiB/s, 4 objects/s recovering 2024-04-02T04:51:20.331 INFO:teuthology.orchestra.run.smithi053.stdout: 5 221363 10.38 MB/sec execute 390 sec latency 39.687 ms 2024-04-02T04:51:20.779 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:20 smithi053 ceph-mon[67180]: failed to encode map e98 with expected crc 2024-04-02T04:51:20.779 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:20 smithi053 ceph-mon[67180]: failed to encode map e98 with expected crc 2024-04-02T04:51:20.779 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:20 smithi053 ceph-mon[67180]: osdmap e98: 8 total, 8 up, 8 in 2024-04-02T04:51:20.779 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:20 smithi053 ceph-mon[67180]: failed to encode map e98 with expected crc 2024-04-02T04:51:20.779 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:20 smithi053 ceph-mon[67180]: failed to encode map e98 with expected crc 2024-04-02T04:51:20.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:20 smithi122 ceph-mon[53211]: failed to encode map e98 with expected crc 2024-04-02T04:51:20.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:20 smithi122 ceph-mon[53211]: failed to encode map e98 with expected crc 2024-04-02T04:51:20.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:20 smithi122 ceph-mon[53211]: osdmap e98: 8 total, 8 up, 8 in 2024-04-02T04:51:20.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:20 smithi122 ceph-mon[53211]: failed to encode map e98 with expected crc 2024-04-02T04:51:20.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:20 smithi122 ceph-mon[53211]: failed to encode map e98 with expected crc 2024-04-02T04:51:21.331 INFO:teuthology.orchestra.run.smithi053.stdout: 5 221793 10.36 MB/sec execute 391 sec latency 59.840 ms 2024-04-02T04:51:21.799 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:21 smithi053 ceph-mon[67180]: failed to encode map e99 with expected crc 2024-04-02T04:51:21.799 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:21 smithi053 ceph-mon[67180]: osdmap e99: 8 total, 8 up, 8 in 2024-04-02T04:51:21.799 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:21 smithi053 ceph-mon[67180]: failed to encode map e99 with expected crc 2024-04-02T04:51:21.799 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:21 smithi053 ceph-mon[67180]: failed to encode map e99 with expected crc 2024-04-02T04:51:21.799 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:21 smithi053 ceph-mon[67180]: Health check update: Degraded data redundancy: 52/1560 objects degraded (3.333%), 6 pgs degraded (PG_DEGRADED) 2024-04-02T04:51:21.799 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:21 smithi053 ceph-mon[67180]: pgmap v138: 97 pgs: 3 peering, 3 activating, 6 remapped+peering, 1 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 9.5 KiB/s rd, 4.0 MiB/s wr, 396 op/s; 25/1560 objects degraded (1.603%); 66 KiB/s, 5 objects/s recovering 2024-04-02T04:51:21.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:21 smithi122 ceph-mon[53211]: failed to encode map e99 with expected crc 2024-04-02T04:51:21.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:21 smithi122 ceph-mon[53211]: osdmap e99: 8 total, 8 up, 8 in 2024-04-02T04:51:21.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:21 smithi122 ceph-mon[53211]: failed to encode map e99 with expected crc 2024-04-02T04:51:21.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:21 smithi122 ceph-mon[53211]: failed to encode map e99 with expected crc 2024-04-02T04:51:21.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:21 smithi122 ceph-mon[53211]: Health check update: Degraded data redundancy: 52/1560 objects degraded (3.333%), 6 pgs degraded (PG_DEGRADED) 2024-04-02T04:51:21.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:21 smithi122 ceph-mon[53211]: pgmap v138: 97 pgs: 3 peering, 3 activating, 6 remapped+peering, 1 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 9.5 KiB/s rd, 4.0 MiB/s wr, 396 op/s; 25/1560 objects degraded (1.603%); 66 KiB/s, 5 objects/s recovering 2024-04-02T04:51:22.331 INFO:teuthology.orchestra.run.smithi053.stdout: 5 222083 10.35 MB/sec execute 392 sec latency 533.410 ms 2024-04-02T04:51:22.844 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:22 smithi053 ceph-mon[67180]: failed to encode map e100 with expected crc 2024-04-02T04:51:22.844 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:22 smithi053 ceph-mon[67180]: failed to encode map e100 with expected crc 2024-04-02T04:51:22.844 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:22 smithi053 ceph-mon[67180]: osdmap e100: 8 total, 8 up, 8 in 2024-04-02T04:51:22.844 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:22 smithi053 ceph-mon[67180]: failed to encode map e100 with expected crc 2024-04-02T04:51:22.844 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:22 smithi053 ceph-mon[67180]: failed to encode map e100 with expected crc 2024-04-02T04:51:22.844 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:22 smithi053 ceph-mon[67180]: failed to encode map e100 with expected crc 2024-04-02T04:51:22.844 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:22 smithi053 ceph-mon[67180]: failed to encode map e100 with expected crc 2024-04-02T04:51:22.844 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:22 smithi053 ceph-mon[67180]: failed to encode map e100 with expected crc 2024-04-02T04:51:22.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:22 smithi122 ceph-mon[53211]: failed to encode map e100 with expected crc 2024-04-02T04:51:22.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:22 smithi122 ceph-mon[53211]: failed to encode map e100 with expected crc 2024-04-02T04:51:22.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:22 smithi122 ceph-mon[53211]: osdmap e100: 8 total, 8 up, 8 in 2024-04-02T04:51:22.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:22 smithi122 ceph-mon[53211]: failed to encode map e100 with expected crc 2024-04-02T04:51:22.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:22 smithi122 ceph-mon[53211]: failed to encode map e100 with expected crc 2024-04-02T04:51:22.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:22 smithi122 ceph-mon[53211]: failed to encode map e100 with expected crc 2024-04-02T04:51:22.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:22 smithi122 ceph-mon[53211]: failed to encode map e100 with expected crc 2024-04-02T04:51:22.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:22 smithi122 ceph-mon[53211]: failed to encode map e100 with expected crc 2024-04-02T04:51:23.331 INFO:teuthology.orchestra.run.smithi053.stdout: 5 222214 10.33 MB/sec execute 393 sec latency 838.397 ms 2024-04-02T04:51:23.875 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:23 smithi053 ceph-mon[67180]: failed to encode map e101 with expected crc 2024-04-02T04:51:23.875 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:23 smithi053 ceph-mon[67180]: osdmap e101: 8 total, 8 up, 8 in 2024-04-02T04:51:23.875 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:23 smithi053 ceph-mon[67180]: failed to encode map e101 with expected crc 2024-04-02T04:51:23.875 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:23 smithi053 ceph-mon[67180]: failed to encode map e101 with expected crc 2024-04-02T04:51:23.875 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:23 smithi053 ceph-mon[67180]: failed to encode map e101 with expected crc 2024-04-02T04:51:23.876 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:23 smithi053 ceph-mon[67180]: failed to encode map e101 with expected crc 2024-04-02T04:51:23.876 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:23 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:23.876 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:23 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:23.876 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:23 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:51:23.876 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:23 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:51:23.876 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:23 smithi053 ceph-mon[67180]: pgmap v141: 97 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 3.0 MiB/s wr, 426 op/s; 37/1551 objects degraded (2.386%); 2.1 MiB/s, 34 objects/s recovering 2024-04-02T04:51:23.876 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:23 smithi053 ceph-mon[67180]: pgmap v142: 97 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 3.3 MiB/s wr, 460 op/s; 37/1551 objects degraded (2.386%); 2.3 MiB/s, 36 objects/s recovering 2024-04-02T04:51:23.876 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:23 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:23.876 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:23 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:23.876 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:23 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:51:23.876 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:23 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:51:23.876 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:23 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:51:23.876 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:23 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:51:23.876 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:23 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-02T04:51:23.877 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:23 smithi053 ceph-mon[67180]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-02T04:51:23.877 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:23 smithi053 ceph-mon[67180]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-04-02T04:51:23.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:23 smithi122 ceph-mon[53211]: failed to encode map e101 with expected crc 2024-04-02T04:51:23.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:23 smithi122 ceph-mon[53211]: osdmap e101: 8 total, 8 up, 8 in 2024-04-02T04:51:23.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:23 smithi122 ceph-mon[53211]: failed to encode map e101 with expected crc 2024-04-02T04:51:23.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:23 smithi122 ceph-mon[53211]: failed to encode map e101 with expected crc 2024-04-02T04:51:23.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:23 smithi122 ceph-mon[53211]: failed to encode map e101 with expected crc 2024-04-02T04:51:23.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:23 smithi122 ceph-mon[53211]: failed to encode map e101 with expected crc 2024-04-02T04:51:23.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:23 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:23.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:23 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:23.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:23 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:51:23.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:23 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:51:23.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:23 smithi122 ceph-mon[53211]: pgmap v141: 97 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 3.0 MiB/s wr, 426 op/s; 37/1551 objects degraded (2.386%); 2.1 MiB/s, 34 objects/s recovering 2024-04-02T04:51:23.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:23 smithi122 ceph-mon[53211]: pgmap v142: 97 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 3.3 MiB/s wr, 460 op/s; 37/1551 objects degraded (2.386%); 2.3 MiB/s, 36 objects/s recovering 2024-04-02T04:51:23.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:23 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:23.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:23 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:23.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:23 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:51:23.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:23 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:51:23.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:23 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:51:23.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:23 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:51:23.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:23 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-02T04:51:23.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:23 smithi122 ceph-mon[53211]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-02T04:51:23.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:23 smithi122 ceph-mon[53211]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-04-02T04:51:24.331 INFO:teuthology.orchestra.run.smithi053.stdout: 5 222444 10.33 MB/sec execute 394 sec latency 104.723 ms 2024-04-02T04:51:24.639 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:24 smithi053 ceph-mon[67180]: failed to encode map e102 with expected crc 2024-04-02T04:51:24.639 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:24 smithi053 ceph-mon[67180]: osdmap e102: 8 total, 8 up, 8 in 2024-04-02T04:51:24.639 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:24 smithi053 ceph-mon[67180]: failed to encode map e102 with expected crc 2024-04-02T04:51:24.639 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:24 smithi053 ceph-mon[67180]: failed to encode map e102 with expected crc 2024-04-02T04:51:24.639 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:24 smithi053 ceph-mon[67180]: failed to encode map e102 with expected crc 2024-04-02T04:51:24.639 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:24 smithi053 ceph-mon[67180]: failed to encode map e102 with expected crc 2024-04-02T04:51:24.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:24 smithi122 ceph-mon[53211]: failed to encode map e102 with expected crc 2024-04-02T04:51:24.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:24 smithi122 ceph-mon[53211]: osdmap e102: 8 total, 8 up, 8 in 2024-04-02T04:51:24.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:24 smithi122 ceph-mon[53211]: failed to encode map e102 with expected crc 2024-04-02T04:51:24.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:24 smithi122 ceph-mon[53211]: failed to encode map e102 with expected crc 2024-04-02T04:51:24.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:24 smithi122 ceph-mon[53211]: failed to encode map e102 with expected crc 2024-04-02T04:51:24.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:24 smithi122 ceph-mon[53211]: failed to encode map e102 with expected crc 2024-04-02T04:51:25.331 INFO:teuthology.orchestra.run.smithi053.stdout: 5 222756 10.34 MB/sec execute 395 sec latency 55.657 ms 2024-04-02T04:51:25.905 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:25 smithi053 ceph-mon[67180]: failed to encode map e103 with expected crc 2024-04-02T04:51:25.905 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:25 smithi053 ceph-mon[67180]: osdmap e103: 8 total, 8 up, 8 in 2024-04-02T04:51:25.905 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:25 smithi053 ceph-mon[67180]: failed to encode map e103 with expected crc 2024-04-02T04:51:25.905 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:25 smithi053 ceph-mon[67180]: pgmap v145: 97 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 94 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.8 MiB/s wr, 493 op/s; 12/1560 objects degraded (0.769%); 2.3 MiB/s, 43 objects/s recovering 2024-04-02T04:51:26.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:25 smithi122 ceph-mon[53211]: failed to encode map e103 with expected crc 2024-04-02T04:51:26.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:25 smithi122 ceph-mon[53211]: osdmap e103: 8 total, 8 up, 8 in 2024-04-02T04:51:26.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:25 smithi122 ceph-mon[53211]: failed to encode map e103 with expected crc 2024-04-02T04:51:26.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:25 smithi122 ceph-mon[53211]: pgmap v145: 97 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 94 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.8 MiB/s wr, 493 op/s; 12/1560 objects degraded (0.769%); 2.3 MiB/s, 43 objects/s recovering 2024-04-02T04:51:26.331 INFO:teuthology.orchestra.run.smithi053.stdout: 5 223050 10.33 MB/sec execute 396 sec latency 79.746 ms 2024-04-02T04:51:26.673 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:26 smithi053 ceph-mon[67180]: Health check update: Degraded data redundancy: 12/1560 objects degraded (0.769%), 1 pg degraded (PG_DEGRADED) 2024-04-02T04:51:26.673 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:26 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:26.673 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:26 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:51:27.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:26 smithi122 ceph-mon[53211]: Health check update: Degraded data redundancy: 12/1560 objects degraded (0.769%), 1 pg degraded (PG_DEGRADED) 2024-04-02T04:51:27.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:26 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:27.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:26 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:51:27.331 INFO:teuthology.orchestra.run.smithi053.stdout: 5 223311 10.33 MB/sec execute 397 sec latency 66.155 ms 2024-04-02T04:51:27.699 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:27 smithi053 ceph-mon[67180]: pgmap v146: 97 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 94 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 538 KiB/s wr, 45 op/s; 12/1560 objects degraded (0.769%); 33 KiB/s, 10 objects/s recovering 2024-04-02T04:51:28.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:27 smithi122 ceph-mon[53211]: pgmap v146: 97 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 94 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 538 KiB/s wr, 45 op/s; 12/1560 objects degraded (0.769%); 33 KiB/s, 10 objects/s recovering 2024-04-02T04:51:28.331 INFO:teuthology.orchestra.run.smithi053.stdout: 5 223545 10.32 MB/sec execute 398 sec latency 58.804 ms 2024-04-02T04:51:29.248 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:29 smithi053 ceph-mon[67180]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/1560 objects degraded (0.769%), 1 pg degraded) 2024-04-02T04:51:29.331 INFO:teuthology.orchestra.run.smithi053.stdout: 5 223807 10.31 MB/sec execute 399 sec latency 89.669 ms 2024-04-02T04:51:29.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:29 smithi122 ceph-mon[53211]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/1560 objects degraded (0.769%), 1 pg degraded) 2024-04-02T04:51:30.262 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:30 smithi053 ceph-mon[67180]: pgmap v147: 97 pgs: 97 active+clean; 345 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.4 MiB/s wr, 308 op/s; 20 KiB/s, 7 objects/s recovering 2024-04-02T04:51:30.331 INFO:teuthology.orchestra.run.smithi053.stdout: 5 224185 10.30 MB/sec execute 400 sec latency 47.117 ms 2024-04-02T04:51:30.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:30 smithi122 ceph-mon[53211]: pgmap v147: 97 pgs: 97 active+clean; 345 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.4 MiB/s wr, 308 op/s; 20 KiB/s, 7 objects/s recovering 2024-04-02T04:51:31.331 INFO:teuthology.orchestra.run.smithi053.stdout: 5 224514 10.30 MB/sec execute 401 sec latency 44.684 ms 2024-04-02T04:51:32.331 INFO:teuthology.orchestra.run.smithi053.stdout: 5 224901 10.30 MB/sec execute 402 sec latency 51.789 ms 2024-04-02T04:51:32.379 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:32 smithi053 ceph-mon[67180]: pgmap v148: 97 pgs: 97 active+clean; 344 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 3.1 MiB/s wr, 250 op/s; 15 KiB/s, 5 objects/s recovering 2024-04-02T04:51:32.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:32 smithi122 ceph-mon[53211]: pgmap v148: 97 pgs: 97 active+clean; 344 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 3.1 MiB/s wr, 250 op/s; 15 KiB/s, 5 objects/s recovering 2024-04-02T04:51:33.331 INFO:teuthology.orchestra.run.smithi053.stdout: 5 225356 10.29 MB/sec execute 403 sec latency 46.416 ms 2024-04-02T04:51:34.331 INFO:teuthology.orchestra.run.smithi053.stdout: 5 225758 10.29 MB/sec execute 404 sec latency 30.868 ms 2024-04-02T04:51:34.406 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:34 smithi053 ceph-mon[67180]: pgmap v149: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.8 MiB/s wr, 463 op/s; 13 KiB/s, 4 objects/s recovering 2024-04-02T04:51:34.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:34 smithi122 ceph-mon[53211]: pgmap v149: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.8 MiB/s wr, 463 op/s; 13 KiB/s, 4 objects/s recovering 2024-04-02T04:51:35.331 INFO:teuthology.orchestra.run.smithi053.stdout: 5 226149 10.30 MB/sec execute 405 sec latency 38.816 ms 2024-04-02T04:51:36.331 INFO:teuthology.orchestra.run.smithi053.stdout: 5 226510 10.31 MB/sec execute 406 sec latency 67.573 ms 2024-04-02T04:51:36.428 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:36 smithi053 ceph-mon[67180]: pgmap v150: 97 pgs: 97 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.4 MiB/s wr, 426 op/s; 0 B/s, 1 objects/s recovering 2024-04-02T04:51:36.614 INFO:teuthology.orchestra.run.smithi053.stdout:true 2024-04-02T04:51:36.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:36 smithi122 ceph-mon[53211]: pgmap v150: 97 pgs: 97 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.4 MiB/s wr, 426 op/s; 0 B/s, 1 objects/s recovering 2024-04-02T04:51:37.331 INFO:teuthology.orchestra.run.smithi053.stdout: 5 226886 10.31 MB/sec execute 407 sec latency 82.331 ms 2024-04-02T04:51:37.538 INFO:teuthology.orchestra.run.smithi053.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-02T04:51:37.539 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager.smithi053 smithi053 *:9093,9094 running (6m) 21s ago 15m 24.3M - 0.23.0 ba2b418f427c bf50c1cd2b3d 2024-04-02T04:51:37.539 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi053 smithi053 running (2m) 21s ago 15m 6660k - 19.0.0-1979-g6cec793d 78f830190839 4ce8915e0157 2024-04-02T04:51:37.539 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi122 smithi122 running (2m) 2m ago 14m 6660k - 19.0.0-1979-g6cec793d 78f830190839 e2b5369e6fdf 2024-04-02T04:51:37.539 INFO:teuthology.orchestra.run.smithi053.stdout:grafana.smithi053 smithi053 *:3000 error 21s ago 14m - - 2024-04-02T04:51:37.539 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi053.nfrfot smithi053 running (9m) 21s ago 9m 37.2M - 17.2.0 e1d6a67b021e c94e9e643514 2024-04-02T04:51:37.539 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi122.mwoyoh smithi122 running (9m) 2m ago 9m 15.6M - 17.2.0 e1d6a67b021e 73376d9ee4b8 2024-04-02T04:51:37.539 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi053.legdmv smithi053 *:8443,9283,8765 running (5m) 21s ago 16m 505M - 19.0.0-1979-g6cec793d 78f830190839 c12d516a31bc 2024-04-02T04:51:37.539 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi122.cpwlyk smithi122 *:8443,9283,8765 running (4m) 2m ago 14m 449M - 19.0.0-1979-g6cec793d 78f830190839 9a545c5e7d81 2024-04-02T04:51:37.539 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi053 smithi053 running (3m) 21s ago 16m 49.5M 2048M 19.0.0-1979-g6cec793d 78f830190839 f53a44f19c00 2024-04-02T04:51:37.539 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi122 smithi122 running (3m) 2m ago 14m 34.8M 2048M 19.0.0-1979-g6cec793d 78f830190839 44b200fed3c5 2024-04-02T04:51:37.539 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.0.0.smithi053.mhkdcb smithi053 *:2049 running (9m) 21s ago 9m 289M - 4.0 e1d6a67b021e 07e5da7bd2c5 2024-04-02T04:51:37.539 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.1.0.smithi122.mauvsb smithi122 *:2049 running (8m) 2m ago 8m 81.7M - 4.0 e1d6a67b021e 4f866685005e 2024-04-02T04:51:37.539 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi053 smithi053 *:9100 running (5m) 21s ago 14m 18.2M - 1.3.1 1dbe0e931976 773b71971e67 2024-04-02T04:51:37.539 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi122 smithi122 *:9100 running (5m) 2m ago 14m 15.7M - 1.3.1 1dbe0e931976 475d439c7980 2024-04-02T04:51:37.539 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 smithi053 running (111s) 21s ago 13m 278M 1937M 19.0.0-1979-g6cec793d 78f830190839 490c346c7639 2024-04-02T04:51:37.539 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 smithi053 running (77s) 21s ago 13m 226M 1937M 19.0.0-1979-g6cec793d 78f830190839 e70e40949c2f 2024-04-02T04:51:37.540 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 smithi053 running (29s) 21s ago 12m 12.7M 1937M 19.0.0-1979-g6cec793d 78f830190839 4f053827e0dd 2024-04-02T04:51:37.540 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 smithi053 running (12m) 21s ago 12m 402M 1937M 17.2.0 e1d6a67b021e 7e8b9e98ae2b 2024-04-02T04:51:37.540 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 smithi122 running (11m) 2m ago 11m 557M 2704M 17.2.0 e1d6a67b021e 46076afcd7bc 2024-04-02T04:51:37.540 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 smithi122 running (11m) 2m ago 11m 389M 2704M 17.2.0 e1d6a67b021e e687a2f6da4d 2024-04-02T04:51:37.540 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 smithi122 running (11m) 2m ago 11m 568M 2704M 17.2.0 e1d6a67b021e 9e371df9a839 2024-04-02T04:51:37.540 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 smithi122 running (10m) 2m ago 10m 480M 2704M 17.2.0 e1d6a67b021e 9b3cc398254d 2024-04-02T04:51:37.540 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus.smithi053 smithi053 *:9095 running (4m) 21s ago 14m 69.6M - 2.33.4 514e6a882f6e 1dd55399da10 2024-04-02T04:51:37.963 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:37 smithi053 ceph-mon[67180]: from='client.34282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:51:37.963 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:37 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:37.963 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:37 smithi053 ceph-mon[67180]: pgmap v151: 97 pgs: 97 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 3.9 MiB/s wr, 373 op/s; 0 B/s, 1 objects/s recovering 2024-04-02T04:51:37.963 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:37 smithi053 ceph-mon[67180]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:51:37.963 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:37 smithi053 ceph-mon[67180]: from='client.34290 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:51:38.049 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:51:38.049 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": { 2024-04-02T04:51:38.049 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:51:38.049 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:51:38.049 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": { 2024-04-02T04:51:38.049 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:51:38.049 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:51:38.049 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": { 2024-04-02T04:51:38.049 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-04-02T04:51:38.050 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 3 2024-04-02T04:51:38.050 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:51:38.050 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": { 2024-04-02T04:51:38.050 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:51:38.050 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:51:38.050 INFO:teuthology.orchestra.run.smithi053.stdout: "overall": { 2024-04-02T04:51:38.050 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-04-02T04:51:38.050 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 7 2024-04-02T04:51:38.050 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-02T04:51:38.050 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:51:38.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:37 smithi122 ceph-mon[53211]: from='client.34282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:51:38.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:37 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:38.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:37 smithi122 ceph-mon[53211]: pgmap v151: 97 pgs: 97 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 3.9 MiB/s wr, 373 op/s; 0 B/s, 1 objects/s recovering 2024-04-02T04:51:38.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:37 smithi122 ceph-mon[53211]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:51:38.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:37 smithi122 ceph-mon[53211]: from='client.34290 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:51:38.331 INFO:teuthology.orchestra.run.smithi053.stdout: 5 227070 10.30 MB/sec execute 408 sec latency 84.928 ms 2024-04-02T04:51:38.454 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:51:38.454 INFO:teuthology.orchestra.run.smithi053.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae", 2024-04-02T04:51:38.454 INFO:teuthology.orchestra.run.smithi053.stdout: "in_progress": true, 2024-04-02T04:51:38.454 INFO:teuthology.orchestra.run.smithi053.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-02T04:51:38.454 INFO:teuthology.orchestra.run.smithi053.stdout: "services_complete": [ 2024-04-02T04:51:38.454 INFO:teuthology.orchestra.run.smithi053.stdout: "mon", 2024-04-02T04:51:38.454 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr", 2024-04-02T04:51:38.454 INFO:teuthology.orchestra.run.smithi053.stdout: "crash" 2024-04-02T04:51:38.454 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-04-02T04:51:38.454 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": "9/23 daemons upgraded", 2024-04-02T04:51:38.454 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "Currently upgrading osd daemons", 2024-04-02T04:51:38.454 INFO:teuthology.orchestra.run.smithi053.stdout: "is_paused": false 2024-04-02T04:51:38.455 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:51:38.735 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:38 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/2056866993' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:51:38.735 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:38 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-02T04:51:38.735 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:38 smithi053 ceph-mon[67180]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-02T04:51:38.735 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:38 smithi053 ceph-mon[67180]: Upgrade: osd.3 is safe to restart 2024-04-02T04:51:38.735 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:38 smithi053 ceph-mon[67180]: from='client.34298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:51:39.005 INFO:teuthology.orchestra.run.smithi053.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-02T04:51:39.005 INFO:teuthology.orchestra.run.smithi053.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-02T04:51:39.005 INFO:teuthology.orchestra.run.smithi053.stdout: daemon grafana.smithi053 on smithi053 is in error state 2024-04-02T04:51:39.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:38 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/2056866993' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:51:39.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:38 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-02T04:51:39.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:38 smithi122 ceph-mon[53211]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-02T04:51:39.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:38 smithi122 ceph-mon[53211]: Upgrade: osd.3 is safe to restart 2024-04-02T04:51:39.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:38 smithi122 ceph-mon[53211]: from='client.34298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:51:39.331 INFO:teuthology.orchestra.run.smithi053.stdout: 5 227239 10.29 MB/sec execute 409 sec latency 51.612 ms 2024-04-02T04:51:39.748 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:39 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/473806307' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:51:39.748 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:39 smithi053 ceph-mon[67180]: pgmap v152: 97 pgs: 97 active+clean; 352 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 6.0 MiB/s wr, 581 op/s; 0 B/s, 1 objects/s recovering 2024-04-02T04:51:40.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:39 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/473806307' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:51:40.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:39 smithi122 ceph-mon[53211]: pgmap v152: 97 pgs: 97 active+clean; 352 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 6.0 MiB/s wr, 581 op/s; 0 B/s, 1 objects/s recovering 2024-04-02T04:51:40.332 INFO:teuthology.orchestra.run.smithi053.stdout: 5 227452 10.27 MB/sec execute 410 sec latency 72.395 ms 2024-04-02T04:51:41.331 INFO:teuthology.orchestra.run.smithi053.stdout: 5 227709 10.26 MB/sec execute 411 sec latency 83.279 ms 2024-04-02T04:51:41.553 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:41 smithi053 ceph-mon[67180]: Upgrade: Updating osd.3 2024-04-02T04:51:41.553 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:41 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:41.553 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:41 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-02T04:51:41.553 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:41 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:51:41.553 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:41 smithi053 ceph-mon[67180]: Deploying daemon osd.3 on smithi053 2024-04-02T04:51:41.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:41 smithi122 ceph-mon[53211]: Upgrade: Updating osd.3 2024-04-02T04:51:41.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:41 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:41.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:41 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-02T04:51:41.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:41 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:51:41.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:41 smithi122 ceph-mon[53211]: Deploying daemon osd.3 on smithi053 2024-04-02T04:51:42.331 INFO:teuthology.orchestra.run.smithi053.stdout: 5 227977 10.25 MB/sec execute 412 sec latency 71.420 ms 2024-04-02T04:51:42.590 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:42 smithi053 ceph-mon[67180]: pgmap v153: 97 pgs: 97 active+clean; 348 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 4.7 MiB/s wr, 449 op/s 2024-04-02T04:51:42.590 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:42 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:42.590 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:42 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:51:42.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:42 smithi122 ceph-mon[53211]: pgmap v153: 97 pgs: 97 active+clean; 348 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 4.7 MiB/s wr, 449 op/s 2024-04-02T04:51:42.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:42 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:42.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:42 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:51:43.331 INFO:teuthology.orchestra.run.smithi053.stdout: 5 228241 10.24 MB/sec execute 413 sec latency 45.016 ms 2024-04-02T04:51:43.607 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:43 smithi053 ceph-mon[67180]: pgmap v154: 97 pgs: 97 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 5.7 MiB/s wr, 585 op/s 2024-04-02T04:51:43.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:43 smithi122 ceph-mon[53211]: pgmap v154: 97 pgs: 97 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 5.7 MiB/s wr, 585 op/s 2024-04-02T04:51:44.332 INFO:teuthology.orchestra.run.smithi053.stdout: 5 228560 10.24 MB/sec execute 414 sec latency 73.884 ms 2024-04-02T04:51:45.334 INFO:teuthology.orchestra.run.smithi053.stdout: 5 228915 10.22 MB/sec execute 415 sec latency 41.399 ms 2024-04-02T04:51:45.925 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:45 smithi053 systemd[1]: Stopping Ceph osd.3 for 48e16742-f0aa-11ee-b647-cb9ed24678a4... 2024-04-02T04:51:46.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:46 smithi053 ceph-mon[67180]: pgmap v155: 97 pgs: 97 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.1 MiB/s wr, 390 op/s 2024-04-02T04:51:46.332 INFO:teuthology.orchestra.run.smithi053.stdout: 5 229232 10.22 MB/sec execute 416 sec latency 45.063 ms 2024-04-02T04:51:46.453 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:46 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3[46334]: 2024-04-02T04:51:46.182+0000 7fbf1bcd7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T04:51:46.453 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:46 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3[46334]: 2024-04-02T04:51:46.182+0000 7fbf1bcd7700 -1 osd.3 103 *** Got signal Terminated *** 2024-04-02T04:51:46.453 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:46 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3[46334]: 2024-04-02T04:51:46.182+0000 7fbf1bcd7700 -1 osd.3 103 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T04:51:46.597 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:46 smithi122 ceph-mon[53211]: pgmap v155: 97 pgs: 97 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.1 MiB/s wr, 390 op/s 2024-04-02T04:51:47.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:47 smithi053 ceph-mon[67180]: osd.3 marked itself down and dead 2024-04-02T04:51:47.331 INFO:teuthology.orchestra.run.smithi053.stdout: 5 229383 10.21 MB/sec execute 417 sec latency 177.611 ms 2024-04-02T04:51:47.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:47 smithi122 ceph-mon[53211]: osd.3 marked itself down and dead 2024-04-02T04:51:48.154 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:47 smithi053 podman[80388]: 2024-04-02 04:51:47.815912764 +0000 UTC m=+1.890518367 container died 7e8b9e98ae2b4a5f996c230b1806a1d841c1e19d305811086a58500a4a451a45 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, vcs-type=git, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., distribution-scope=public, ceph=True, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, release=754) 2024-04-02T04:51:48.237 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:48 smithi053 ceph-mon[67180]: pgmap v156: 97 pgs: 97 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.9 MiB/s wr, 390 op/s 2024-04-02T04:51:48.237 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:48 smithi053 ceph-mon[67180]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-02T04:51:48.237 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:48 smithi053 ceph-mon[67180]: failed to encode map e104 with expected crc 2024-04-02T04:51:48.237 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:48 smithi053 ceph-mon[67180]: osdmap e104: 8 total, 7 up, 8 in 2024-04-02T04:51:48.238 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:48 smithi053 ceph-mon[67180]: failed to encode map e104 with expected crc 2024-04-02T04:51:48.238 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:48 smithi053 ceph-mon[67180]: failed to encode map e104 with expected crc 2024-04-02T04:51:48.238 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:48 smithi053 ceph-mon[67180]: failed to encode map e104 with expected crc 2024-04-02T04:51:48.238 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:48 smithi053 ceph-mon[67180]: failed to encode map e104 with expected crc 2024-04-02T04:51:48.331 INFO:teuthology.orchestra.run.smithi053.stdout: 5 229442 10.18 MB/sec execute 418 sec latency 1039.781 ms 2024-04-02T04:51:48.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:48 smithi122 ceph-mon[53211]: pgmap v156: 97 pgs: 97 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.9 MiB/s wr, 390 op/s 2024-04-02T04:51:48.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:48 smithi122 ceph-mon[53211]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-02T04:51:48.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:48 smithi122 ceph-mon[53211]: failed to encode map e104 with expected crc 2024-04-02T04:51:48.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:48 smithi122 ceph-mon[53211]: osdmap e104: 8 total, 7 up, 8 in 2024-04-02T04:51:48.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:48 smithi122 ceph-mon[53211]: failed to encode map e104 with expected crc 2024-04-02T04:51:48.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:48 smithi122 ceph-mon[53211]: failed to encode map e104 with expected crc 2024-04-02T04:51:48.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:48 smithi122 ceph-mon[53211]: failed to encode map e104 with expected crc 2024-04-02T04:51:48.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:48 smithi122 ceph-mon[53211]: failed to encode map e104 with expected crc 2024-04-02T04:51:48.916 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:48 smithi053 podman[80388]: 2024-04-02 04:51:48.750266614 +0000 UTC m=+2.824872229 container cleanup 7e8b9e98ae2b4a5f996c230b1806a1d841c1e19d305811086a58500a4a451a45 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, ceph=True, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, GIT_BRANCH=HEAD, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, RELEASE=HEAD, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-02T04:51:48.916 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:48 smithi053 bash[80388]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3 2024-04-02T04:51:49.332 INFO:teuthology.orchestra.run.smithi053.stdout: 5 229877 10.21 MB/sec execute 419 sec latency 37.555 ms 2024-04-02T04:51:49.430 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:49 smithi053 podman[80404]: 2024-04-02 04:51:49.27278058 +0000 UTC m=+1.449122542 container remove 7e8b9e98ae2b4a5f996c230b1806a1d841c1e19d305811086a58500a4a451a45 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3, GIT_CLEAN=True, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, io.buildah.version=1.19.8, RELEASE=HEAD, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, io.openshift.expose-services=, vendor=Red Hat, Inc., distribution-scope=public, name=centos-stream, vcs-type=git, ceph=True, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-02T04:51:49.430 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:49 smithi053 ceph-mon[67180]: failed to encode map e105 with expected crc 2024-04-02T04:51:49.430 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:49 smithi053 ceph-mon[67180]: osdmap e105: 8 total, 7 up, 8 in 2024-04-02T04:51:49.430 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:49 smithi053 ceph-mon[67180]: failed to encode map e105 with expected crc 2024-04-02T04:51:49.431 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:49 smithi053 ceph-mon[67180]: failed to encode map e105 with expected crc 2024-04-02T04:51:49.431 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:49 smithi053 ceph-mon[67180]: failed to encode map e105 with expected crc 2024-04-02T04:51:49.431 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:49 smithi053 ceph-mon[67180]: failed to encode map e105 with expected crc 2024-04-02T04:51:49.431 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:49 smithi053 ceph-mon[67180]: failed to encode map e105 with expected crc 2024-04-02T04:51:49.431 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:49 smithi053 ceph-mon[67180]: failed to encode map e105 with expected crc 2024-04-02T04:51:49.431 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:49 smithi053 ceph-mon[67180]: failed to encode map e105 with expected crc 2024-04-02T04:51:49.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:49 smithi122 ceph-mon[53211]: failed to encode map e105 with expected crc 2024-04-02T04:51:49.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:49 smithi122 ceph-mon[53211]: osdmap e105: 8 total, 7 up, 8 in 2024-04-02T04:51:49.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:49 smithi122 ceph-mon[53211]: failed to encode map e105 with expected crc 2024-04-02T04:51:49.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:49 smithi122 ceph-mon[53211]: failed to encode map e105 with expected crc 2024-04-02T04:51:49.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:49 smithi122 ceph-mon[53211]: failed to encode map e105 with expected crc 2024-04-02T04:51:49.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:49 smithi122 ceph-mon[53211]: failed to encode map e105 with expected crc 2024-04-02T04:51:49.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:49 smithi122 ceph-mon[53211]: failed to encode map e105 with expected crc 2024-04-02T04:51:49.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:49 smithi122 ceph-mon[53211]: failed to encode map e105 with expected crc 2024-04-02T04:51:49.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:49 smithi122 ceph-mon[53211]: failed to encode map e105 with expected crc 2024-04-02T04:51:49.939 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:49 smithi053 podman[80518]: 2024-04-02 04:51:49.795113736 +0000 UTC m=+0.027423819 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:51:50.190 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:50 smithi053 ceph-mon[67180]: pgmap v159: 97 pgs: 19 peering, 78 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.0 MiB/s wr, 415 op/s 2024-04-02T04:51:50.190 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:50 smithi053 ceph-mon[67180]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-04-02T04:51:50.190 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:49 smithi053 podman[80518]: 2024-04-02 04:51:49.93796907 +0000 UTC m=+0.170279170 container create 155dfc02171593fef158cf0846605dc025104e2c1114d2007c0001af7f55623d (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-6cec793) 2024-04-02T04:51:50.332 INFO:teuthology.orchestra.run.smithi053.stdout: 5 230307 10.22 MB/sec execute 420 sec latency 45.317 ms 2024-04-02T04:51:50.441 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:50 smithi053 podman[80518]: 2024-04-02 04:51:50.325637241 +0000 UTC m=+0.557947316 container init 155dfc02171593fef158cf0846605dc025104e2c1114d2007c0001af7f55623d (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-6cec793, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T04:51:50.441 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:50 smithi053 podman[80518]: 2024-04-02 04:51:50.33341945 +0000 UTC m=+0.565729517 container start 155dfc02171593fef158cf0846605dc025104e2c1114d2007c0001af7f55623d (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-6cec793, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326) 2024-04-02T04:51:50.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:50 smithi122 ceph-mon[53211]: pgmap v159: 97 pgs: 19 peering, 78 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.0 MiB/s wr, 415 op/s 2024-04-02T04:51:50.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:50 smithi122 ceph-mon[53211]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-04-02T04:51:50.691 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:50 smithi053 podman[80518]: 2024-04-02 04:51:50.660878262 +0000 UTC m=+0.893188326 container attach 155dfc02171593fef158cf0846605dc025104e2c1114d2007c0001af7f55623d (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, RELEASE=squid-6cec793, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=) 2024-04-02T04:51:50.948 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:50 smithi053 podman[80518]: 2024-04-02 04:51:50.809849519 +0000 UTC m=+1.042159580 container died 155dfc02171593fef158cf0846605dc025104e2c1114d2007c0001af7f55623d (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3-deactivate, RELEASE=squid-6cec793, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True) 2024-04-02T04:51:51.332 INFO:teuthology.orchestra.run.smithi053.stdout: 5 230683 10.22 MB/sec execute 421 sec latency 34.447 ms 2024-04-02T04:51:51.466 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:51 smithi053 ceph-mon[67180]: Health check failed: Degraded data redundancy: 77/1503 objects degraded (5.123%), 10 pgs degraded (PG_DEGRADED) 2024-04-02T04:51:51.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:51 smithi122 ceph-mon[53211]: Health check failed: Degraded data redundancy: 77/1503 objects degraded (5.123%), 10 pgs degraded (PG_DEGRADED) 2024-04-02T04:51:52.234 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:52 smithi053 ceph-mon[67180]: pgmap v160: 97 pgs: 4 active+undersized, 10 active+undersized+degraded, 19 peering, 64 active+clean; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 2.8 MiB/s wr, 340 op/s; 77/1503 objects degraded (5.123%) 2024-04-02T04:51:52.234 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:52 smithi053 podman[80543]: 2024-04-02 04:51:52.031159573 +0000 UTC m=+1.214367100 container remove 155dfc02171593fef158cf0846605dc025104e2c1114d2007c0001af7f55623d (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3-deactivate, RELEASE=squid-6cec793, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-02T04:51:52.234 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:52 smithi053 systemd[1]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.3.service: Deactivated successfully. 2024-04-02T04:51:52.234 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:52 smithi053 systemd[1]: Stopped Ceph osd.3 for 48e16742-f0aa-11ee-b647-cb9ed24678a4. 2024-04-02T04:51:52.234 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:52 smithi053 systemd[1]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.3.service: Consumed 4min 25.956s CPU time. 2024-04-02T04:51:52.332 INFO:teuthology.orchestra.run.smithi053.stdout: 5 230967 10.21 MB/sec execute 422 sec latency 38.129 ms 2024-04-02T04:51:52.485 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:52 smithi053 systemd[1]: Starting Ceph osd.3 for 48e16742-f0aa-11ee-b647-cb9ed24678a4... 2024-04-02T04:51:52.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:52 smithi122 ceph-mon[53211]: pgmap v160: 97 pgs: 4 active+undersized, 10 active+undersized+degraded, 19 peering, 64 active+clean; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 2.8 MiB/s wr, 340 op/s; 77/1503 objects degraded (5.123%) 2024-04-02T04:51:52.743 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:52 smithi053 podman[80634]: 2024-04-02 04:51:52.569081456 +0000 UTC m=+0.026234323 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:51:52.743 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:52 smithi053 podman[80634]: 2024-04-02 04:51:52.702599569 +0000 UTC m=+0.159752421 container create 1651bca33a56087bff221954e2a3586223c80c4738335de9b7830a11b1f8424f (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3-activate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-6cec793, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-02T04:51:53.249 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:53 smithi053 podman[80634]: 2024-04-02 04:51:53.041159563 +0000 UTC m=+0.498312417 container init 1651bca33a56087bff221954e2a3586223c80c4738335de9b7830a11b1f8424f (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3-activate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-6cec793) 2024-04-02T04:51:53.249 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:53 smithi053 podman[80634]: 2024-04-02 04:51:53.049845665 +0000 UTC m=+0.506998515 container start 1651bca33a56087bff221954e2a3586223c80c4738335de9b7830a11b1f8424f (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3-activate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-6cec793, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T04:51:53.249 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:53 smithi053 podman[80634]: 2024-04-02 04:51:53.223529059 +0000 UTC m=+0.680681909 container attach 1651bca33a56087bff221954e2a3586223c80c4738335de9b7830a11b1f8424f (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3-activate, RELEASE=squid-6cec793, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux ) 2024-04-02T04:51:53.332 INFO:teuthology.orchestra.run.smithi053.stdout: 5 231277 10.19 MB/sec execute 423 sec latency 35.828 ms 2024-04-02T04:51:53.511 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:53 smithi053 ceph-mon[67180]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-04-02T04:51:53.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:53 smithi122 ceph-mon[53211]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-04-02T04:51:53.766 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:53 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3-activate[80648]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-02T04:51:53.766 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:53 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3-activate[80648]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-02T04:51:53.767 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:53 smithi053 bash[80634]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-02T04:51:53.767 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:53 smithi053 bash[80634]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-02T04:51:53.767 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:53 smithi053 bash[80634]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-02T04:51:53.767 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:53 smithi053 bash[80634]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-02T04:51:53.767 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:53 smithi053 bash[80634]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-04-02T04:51:53.767 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:53 smithi053 bash[80634]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-02T04:51:53.767 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:53 smithi053 bash[80634]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-02T04:51:53.767 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:53 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3-activate[80648]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-02T04:51:53.767 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:53 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3-activate[80648]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-02T04:51:53.767 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:53 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3-activate[80648]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-04-02T04:51:53.767 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:53 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3-activate[80648]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-02T04:51:53.767 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:53 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3-activate[80648]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-02T04:51:53.767 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:53 smithi053 podman[80634]: 2024-04-02 04:51:53.726144652 +0000 UTC m=+1.183300348 container died 1651bca33a56087bff221954e2a3586223c80c4738335de9b7830a11b1f8424f (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3-activate, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-6cec793, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-02T04:51:54.332 INFO:teuthology.orchestra.run.smithi053.stdout: 5 231562 10.19 MB/sec execute 424 sec latency 45.393 ms 2024-04-02T04:51:54.527 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:54 smithi053 ceph-mon[67180]: pgmap v161: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 350 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 5.2 MiB/s wr, 524 op/s; 215/1671 objects degraded (12.867%) 2024-04-02T04:51:54.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:54 smithi122 ceph-mon[53211]: pgmap v161: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 350 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 5.2 MiB/s wr, 524 op/s; 215/1671 objects degraded (12.867%) 2024-04-02T04:51:55.037 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:54 smithi053 podman[80769]: 2024-04-02 04:51:54.822186342 +0000 UTC m=+1.089955201 container remove 1651bca33a56087bff221954e2a3586223c80c4738335de9b7830a11b1f8424f (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3-activate, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-6cec793, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-02T04:51:55.288 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:55 smithi053 podman[80830]: 2024-04-02 04:51:54.984100369 +0000 UTC m=+0.030488965 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:51:55.288 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:55 smithi053 podman[80830]: 2024-04-02 04:51:55.259017417 +0000 UTC m=+0.305406010 container create 5cd6de7a65352d2b97c16dbd45c24243e03b570d833f9889302776e0df666e30 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, RELEASE=squid-6cec793, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-02T04:51:55.332 INFO:teuthology.orchestra.run.smithi053.stdout: 5 231893 10.19 MB/sec execute 425 sec latency 38.382 ms 2024-04-02T04:51:56.308 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:56 smithi053 ceph-mon[67180]: pgmap v162: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 350 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.6 MiB/s wr, 524 op/s; 208/1659 objects degraded (12.538%) 2024-04-02T04:51:56.308 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:56 smithi053 podman[80830]: 2024-04-02 04:51:56.084066943 +0000 UTC m=+1.130455545 container init 5cd6de7a65352d2b97c16dbd45c24243e03b570d833f9889302776e0df666e30 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, RELEASE=squid-6cec793, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T04:51:56.308 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:56 smithi053 podman[80830]: 2024-04-02 04:51:56.095260879 +0000 UTC m=+1.141649472 container start 5cd6de7a65352d2b97c16dbd45c24243e03b570d833f9889302776e0df666e30 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-6cec793, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True) 2024-04-02T04:51:56.308 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:56 smithi053 ceph-osd[80847]: -- 172.21.15.53:0/2133945961 <== mon.0 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d8b5d8f680 con 0x55d8b5d59000 2024-04-02T04:51:56.308 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:56 smithi053 bash[80830]: 5cd6de7a65352d2b97c16dbd45c24243e03b570d833f9889302776e0df666e30 2024-04-02T04:51:56.308 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:56 smithi053 systemd[1]: Started Ceph osd.3 for 48e16742-f0aa-11ee-b647-cb9ed24678a4. 2024-04-02T04:51:56.332 INFO:teuthology.orchestra.run.smithi053.stdout: 5 232224 10.18 MB/sec execute 426 sec latency 71.069 ms 2024-04-02T04:51:56.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:56 smithi122 ceph-mon[53211]: pgmap v162: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 350 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.6 MiB/s wr, 524 op/s; 208/1659 objects degraded (12.538%) 2024-04-02T04:51:56.838 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:51:56 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3[80843]: 2024-04-02T04:51:56.685+0000 7f0e44b9b7c0 -1 Falling back to public interface 2024-04-02T04:51:57.332 INFO:teuthology.orchestra.run.smithi053.stdout: 5 232572 10.16 MB/sec execute 427 sec latency 78.945 ms 2024-04-02T04:51:57.339 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:57 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:57.340 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:57 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:57.340 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:57 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:51:57.340 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:57 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:57.340 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:57 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:51:57.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:57 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:57.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:57 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:57.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:57 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:51:57.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:57 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:51:57.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:57 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:51:58.332 INFO:teuthology.orchestra.run.smithi053.stdout: 5 232862 10.16 MB/sec execute 428 sec latency 36.695 ms 2024-04-02T04:51:58.385 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:58 smithi053 ceph-mon[67180]: pgmap v163: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 350 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.5 MiB/s wr, 423 op/s; 208/1659 objects degraded (12.538%) 2024-04-02T04:51:58.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:58 smithi122 ceph-mon[53211]: pgmap v163: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 350 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.5 MiB/s wr, 423 op/s; 208/1659 objects degraded (12.538%) 2024-04-02T04:51:59.332 INFO:teuthology.orchestra.run.smithi053.stdout: 5 233185 10.17 MB/sec execute 429 sec latency 61.471 ms 2024-04-02T04:51:59.409 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:51:59 smithi053 ceph-mon[67180]: Health check update: Degraded data redundancy: 193/1560 objects degraded (12.372%), 27 pgs degraded (PG_DEGRADED) 2024-04-02T04:51:59.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:51:59 smithi122 ceph-mon[53211]: Health check update: Degraded data redundancy: 193/1560 objects degraded (12.372%), 27 pgs degraded (PG_DEGRADED) 2024-04-02T04:52:00.332 INFO:teuthology.orchestra.run.smithi053.stdout: 5 233533 10.17 MB/sec execute 430 sec latency 69.792 ms 2024-04-02T04:52:00.429 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:00 smithi053 ceph-mon[67180]: pgmap v164: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.3 MiB/s wr, 464 op/s; 193/1560 objects degraded (12.372%) 2024-04-02T04:52:00.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:00 smithi122 ceph-mon[53211]: pgmap v164: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.3 MiB/s wr, 464 op/s; 193/1560 objects degraded (12.372%) 2024-04-02T04:52:01.332 INFO:teuthology.orchestra.run.smithi053.stdout: 5 233862 10.17 MB/sec execute 431 sec latency 45.860 ms 2024-04-02T04:52:01.949 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:52:01 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3[80843]: 2024-04-02T04:52:01.896+0000 7f0e44b9b7c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-04-02T04:52:02.332 INFO:teuthology.orchestra.run.smithi053.stdout: 5 234097 10.17 MB/sec execute 432 sec latency 55.453 ms 2024-04-02T04:52:02.465 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:02 smithi053 ceph-mon[67180]: pgmap v165: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.2 MiB/s wr, 442 op/s; 196/1578 objects degraded (12.421%) 2024-04-02T04:52:02.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:02 smithi122 ceph-mon[53211]: pgmap v165: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.2 MiB/s wr, 442 op/s; 196/1578 objects degraded (12.421%) 2024-04-02T04:52:03.332 INFO:teuthology.orchestra.run.smithi053.stdout: 5 234272 10.16 MB/sec execute 433 sec latency 67.353 ms 2024-04-02T04:52:03.494 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:03 smithi053 ceph-mon[67180]: from='osd.3 [v2:172.21.15.53:6826/2402943060,v1:172.21.15.53:6827/2402943060]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-02T04:52:03.494 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:52:03 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3[80843]: 2024-04-02T04:52:03.367+0000 7f0e44b9b7c0 -1 osd.3 103 log_to_monitors true 2024-04-02T04:52:03.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:03 smithi122 ceph-mon[53211]: from='osd.3 [v2:172.21.15.53:6826/2402943060,v1:172.21.15.53:6827/2402943060]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-02T04:52:03.751 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:52:03 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3[80843]: 2024-04-02T04:52:03.719+0000 7f0e31feb700 -1 osd.3 103 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T04:52:04.332 INFO:teuthology.orchestra.run.smithi053.stdout: 5 234584 10.15 MB/sec execute 434 sec latency 48.980 ms 2024-04-02T04:52:04.513 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:04 smithi053 ceph-mon[67180]: pgmap v166: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 344 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 5.4 MiB/s wr, 505 op/s; 206/1620 objects degraded (12.716%) 2024-04-02T04:52:04.513 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:04 smithi053 ceph-mon[67180]: from='osd.3 [v2:172.21.15.53:6826/2402943060,v1:172.21.15.53:6827/2402943060]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-02T04:52:04.513 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:04 smithi053 ceph-mon[67180]: failed to encode map e106 with expected crc 2024-04-02T04:52:04.513 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:04 smithi053 ceph-mon[67180]: osdmap e106: 8 total, 7 up, 8 in 2024-04-02T04:52:04.513 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:04 smithi053 ceph-mon[67180]: from='osd.3 [v2:172.21.15.53:6826/2402943060,v1:172.21.15.53:6827/2402943060]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-02T04:52:04.513 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:04 smithi053 ceph-mon[67180]: from='osd.3 [v2:172.21.15.53:6826/2402943060,v1:172.21.15.53:6827/2402943060]' entity='osd.3' 2024-04-02T04:52:04.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:04 smithi122 ceph-mon[53211]: pgmap v166: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 344 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 5.4 MiB/s wr, 505 op/s; 206/1620 objects degraded (12.716%) 2024-04-02T04:52:04.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:04 smithi122 ceph-mon[53211]: from='osd.3 [v2:172.21.15.53:6826/2402943060,v1:172.21.15.53:6827/2402943060]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-02T04:52:04.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:04 smithi122 ceph-mon[53211]: failed to encode map e106 with expected crc 2024-04-02T04:52:04.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:04 smithi122 ceph-mon[53211]: osdmap e106: 8 total, 7 up, 8 in 2024-04-02T04:52:04.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:04 smithi122 ceph-mon[53211]: from='osd.3 [v2:172.21.15.53:6826/2402943060,v1:172.21.15.53:6827/2402943060]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-04-02T04:52:04.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:04 smithi122 ceph-mon[53211]: from='osd.3 [v2:172.21.15.53:6826/2402943060,v1:172.21.15.53:6827/2402943060]' entity='osd.3' 2024-04-02T04:52:05.332 INFO:teuthology.orchestra.run.smithi053.stdout: 5 234788 10.13 MB/sec execute 435 sec latency 605.457 ms 2024-04-02T04:52:05.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:05 smithi122 ceph-mon[53211]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-02T04:52:05.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:05 smithi122 ceph-mon[53211]: failed to encode map e107 with expected crc 2024-04-02T04:52:05.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:05 smithi122 ceph-mon[53211]: osd.3 [v2:172.21.15.53:6826/2402943060,v1:172.21.15.53:6827/2402943060] boot 2024-04-02T04:52:05.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:05 smithi122 ceph-mon[53211]: osdmap e107: 8 total, 8 up, 8 in 2024-04-02T04:52:05.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:05 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:52:05.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:05 smithi053 ceph-mon[67180]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-02T04:52:05.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:05 smithi053 ceph-mon[67180]: failed to encode map e107 with expected crc 2024-04-02T04:52:05.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:05 smithi053 ceph-mon[67180]: osd.3 [v2:172.21.15.53:6826/2402943060,v1:172.21.15.53:6827/2402943060] boot 2024-04-02T04:52:05.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:05 smithi053 ceph-mon[67180]: osdmap e107: 8 total, 8 up, 8 in 2024-04-02T04:52:05.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:05 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T04:52:06.332 INFO:teuthology.orchestra.run.smithi053.stdout: 5 234877 10.11 MB/sec execute 436 sec latency 1290.186 ms 2024-04-02T04:52:06.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:06 smithi122 ceph-mon[53211]: pgmap v169: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 347 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 3.6 KiB/s rd, 5.6 MiB/s wr, 560 op/s; 214/1662 objects degraded (12.876%) 2024-04-02T04:52:06.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:06 smithi122 ceph-mon[53211]: Health check update: Degraded data redundancy: 214/1662 objects degraded (12.876%), 27 pgs degraded (PG_DEGRADED) 2024-04-02T04:52:06.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:06 smithi122 ceph-mon[53211]: failed to encode map e108 with expected crc 2024-04-02T04:52:06.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:06 smithi122 ceph-mon[53211]: osdmap e108: 8 total, 8 up, 8 in 2024-04-02T04:52:06.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:06 smithi122 ceph-mon[53211]: failed to encode map e108 with expected crc 2024-04-02T04:52:06.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:06 smithi122 ceph-mon[53211]: failed to encode map e108 with expected crc 2024-04-02T04:52:06.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:06 smithi122 ceph-mon[53211]: failed to encode map e108 with expected crc 2024-04-02T04:52:06.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:06 smithi122 ceph-mon[53211]: failed to encode map e108 with expected crc 2024-04-02T04:52:06.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:06 smithi122 ceph-mon[53211]: failed to encode map e108 with expected crc 2024-04-02T04:52:06.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:06 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:06.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:06 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:06.752 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:06 smithi053 ceph-mon[67180]: pgmap v169: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 347 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 3.6 KiB/s rd, 5.6 MiB/s wr, 560 op/s; 214/1662 objects degraded (12.876%) 2024-04-02T04:52:06.753 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:06 smithi053 ceph-mon[67180]: Health check update: Degraded data redundancy: 214/1662 objects degraded (12.876%), 27 pgs degraded (PG_DEGRADED) 2024-04-02T04:52:06.753 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:06 smithi053 ceph-mon[67180]: failed to encode map e108 with expected crc 2024-04-02T04:52:06.753 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:06 smithi053 ceph-mon[67180]: osdmap e108: 8 total, 8 up, 8 in 2024-04-02T04:52:06.753 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:06 smithi053 ceph-mon[67180]: failed to encode map e108 with expected crc 2024-04-02T04:52:06.753 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:06 smithi053 ceph-mon[67180]: failed to encode map e108 with expected crc 2024-04-02T04:52:06.753 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:06 smithi053 ceph-mon[67180]: failed to encode map e108 with expected crc 2024-04-02T04:52:06.753 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:06 smithi053 ceph-mon[67180]: failed to encode map e108 with expected crc 2024-04-02T04:52:06.753 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:06 smithi053 ceph-mon[67180]: failed to encode map e108 with expected crc 2024-04-02T04:52:06.753 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:06 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:06.753 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:06 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:07.332 INFO:teuthology.orchestra.run.smithi053.stdout: 5 235075 10.11 MB/sec execute 437 sec latency 1749.294 ms 2024-04-02T04:52:07.515 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:07 smithi053 ceph-mon[67180]: failed to encode map e108 with expected crc 2024-04-02T04:52:07.515 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:07 smithi053 ceph-mon[67180]: failed to encode map e108 with expected crc 2024-04-02T04:52:07.515 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:07 smithi053 ceph-mon[67180]: failed to encode map e109 with expected crc 2024-04-02T04:52:07.515 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:07 smithi053 ceph-mon[67180]: failed to encode map e109 with expected crc 2024-04-02T04:52:07.516 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:07 smithi053 ceph-mon[67180]: osdmap e109: 8 total, 8 up, 8 in 2024-04-02T04:52:07.516 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:07 smithi053 ceph-mon[67180]: failed to encode map e109 with expected crc 2024-04-02T04:52:07.516 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:07 smithi053 ceph-mon[67180]: failed to encode map e109 with expected crc 2024-04-02T04:52:07.516 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:07 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:07.516 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:07 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:07.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:07 smithi122 ceph-mon[53211]: failed to encode map e108 with expected crc 2024-04-02T04:52:07.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:07 smithi122 ceph-mon[53211]: failed to encode map e108 with expected crc 2024-04-02T04:52:07.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:07 smithi122 ceph-mon[53211]: failed to encode map e109 with expected crc 2024-04-02T04:52:07.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:07 smithi122 ceph-mon[53211]: failed to encode map e109 with expected crc 2024-04-02T04:52:07.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:07 smithi122 ceph-mon[53211]: osdmap e109: 8 total, 8 up, 8 in 2024-04-02T04:52:07.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:07 smithi122 ceph-mon[53211]: failed to encode map e109 with expected crc 2024-04-02T04:52:07.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:07 smithi122 ceph-mon[53211]: failed to encode map e109 with expected crc 2024-04-02T04:52:07.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:07 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:07.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:07 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:08.332 INFO:teuthology.orchestra.run.smithi053.stdout: 5 235370 10.10 MB/sec execute 438 sec latency 575.699 ms 2024-04-02T04:52:08.408 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:08 smithi053 ceph-mon[67180]: pgmap v172: 97 pgs: 4 peering, 12 active+undersized, 25 active+undersized+degraded, 56 active+clean; 347 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 1.1 MiB/s wr, 39 op/s; 201/1662 objects degraded (12.094%) 2024-04-02T04:52:08.408 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:08 smithi053 ceph-mon[67180]: osdmap e110: 8 total, 8 up, 8 in 2024-04-02T04:52:08.408 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:08 smithi053 ceph-mon[67180]: failed to encode map e110 with expected crc 2024-04-02T04:52:08.408 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:08 smithi053 ceph-mon[67180]: failed to encode map e110 with expected crc 2024-04-02T04:52:08.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:08 smithi122 ceph-mon[53211]: pgmap v172: 97 pgs: 4 peering, 12 active+undersized, 25 active+undersized+degraded, 56 active+clean; 347 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 1.1 MiB/s wr, 39 op/s; 201/1662 objects degraded (12.094%) 2024-04-02T04:52:08.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:08 smithi122 ceph-mon[53211]: osdmap e110: 8 total, 8 up, 8 in 2024-04-02T04:52:08.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:08 smithi122 ceph-mon[53211]: failed to encode map e110 with expected crc 2024-04-02T04:52:08.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:08 smithi122 ceph-mon[53211]: failed to encode map e110 with expected crc 2024-04-02T04:52:09.332 INFO:teuthology.orchestra.run.smithi053.stdout: 5 235675 10.09 MB/sec execute 439 sec latency 1045.787 ms 2024-04-02T04:52:09.526 INFO:teuthology.orchestra.run.smithi053.stdout:true 2024-04-02T04:52:09.875 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:09 smithi053 ceph-mon[67180]: osdmap e111: 8 total, 8 up, 8 in 2024-04-02T04:52:09.875 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:09 smithi053 ceph-mon[67180]: failed to encode map e111 with expected crc 2024-04-02T04:52:09.875 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:09 smithi053 ceph-mon[67180]: failed to encode map e111 with expected crc 2024-04-02T04:52:09.875 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:09 smithi053 ceph-mon[67180]: failed to encode map e111 with expected crc 2024-04-02T04:52:09.875 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:09 smithi053 ceph-mon[67180]: failed to encode map e111 with expected crc 2024-04-02T04:52:09.875 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:09 smithi053 ceph-mon[67180]: failed to encode map e111 with expected crc 2024-04-02T04:52:09.876 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:09 smithi053 ceph-mon[67180]: failed to encode map e111 with expected crc 2024-04-02T04:52:09.876 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:09 smithi053 ceph-mon[67180]: failed to encode map e111 with expected crc 2024-04-02T04:52:09.876 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:09 smithi053 ceph-mon[67180]: failed to encode map e111 with expected crc 2024-04-02T04:52:09.876 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:09 smithi053 ceph-mon[67180]: failed to encode map e111 with expected crc 2024-04-02T04:52:09.876 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:09 smithi053 ceph-mon[67180]: pgmap v175: 97 pgs: 4 peering, 2 active+undersized, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 85 active+clean; 337 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.4 MiB/s wr, 407 op/s; 52/1530 objects degraded (3.399%); 395 KiB/s, 39 objects/s recovering 2024-04-02T04:52:09.876 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:09 smithi053 ceph-mon[67180]: from='client.44215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:52:10.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:09 smithi122 ceph-mon[53211]: osdmap e111: 8 total, 8 up, 8 in 2024-04-02T04:52:10.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:09 smithi122 ceph-mon[53211]: failed to encode map e111 with expected crc 2024-04-02T04:52:10.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:09 smithi122 ceph-mon[53211]: failed to encode map e111 with expected crc 2024-04-02T04:52:10.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:09 smithi122 ceph-mon[53211]: failed to encode map e111 with expected crc 2024-04-02T04:52:10.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:09 smithi122 ceph-mon[53211]: failed to encode map e111 with expected crc 2024-04-02T04:52:10.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:09 smithi122 ceph-mon[53211]: failed to encode map e111 with expected crc 2024-04-02T04:52:10.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:09 smithi122 ceph-mon[53211]: failed to encode map e111 with expected crc 2024-04-02T04:52:10.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:09 smithi122 ceph-mon[53211]: failed to encode map e111 with expected crc 2024-04-02T04:52:10.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:09 smithi122 ceph-mon[53211]: failed to encode map e111 with expected crc 2024-04-02T04:52:10.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:09 smithi122 ceph-mon[53211]: failed to encode map e111 with expected crc 2024-04-02T04:52:10.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:09 smithi122 ceph-mon[53211]: pgmap v175: 97 pgs: 4 peering, 2 active+undersized, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 85 active+clean; 337 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.4 MiB/s wr, 407 op/s; 52/1530 objects degraded (3.399%); 395 KiB/s, 39 objects/s recovering 2024-04-02T04:52:10.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:09 smithi122 ceph-mon[53211]: from='client.44215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:52:10.332 INFO:teuthology.orchestra.run.smithi053.stdout: 5 235914 10.08 MB/sec execute 440 sec latency 1035.010 ms 2024-04-02T04:52:10.581 INFO:teuthology.orchestra.run.smithi053.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-02T04:52:10.581 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager.smithi053 smithi053 *:9093,9094 running (6m) 4s ago 15m 23.6M - 0.23.0 ba2b418f427c bf50c1cd2b3d 2024-04-02T04:52:10.581 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi053 smithi053 running (3m) 4s ago 15m 6660k - 19.0.0-1979-g6cec793d 78f830190839 4ce8915e0157 2024-04-02T04:52:10.581 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi122 smithi122 running (2m) 2m ago 15m 6660k - 19.0.0-1979-g6cec793d 78f830190839 e2b5369e6fdf 2024-04-02T04:52:10.581 INFO:teuthology.orchestra.run.smithi053.stdout:grafana.smithi053 smithi053 *:3000 error 4s ago 15m - - 2024-04-02T04:52:10.582 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi053.nfrfot smithi053 running (9m) 4s ago 9m 37.4M - 17.2.0 e1d6a67b021e c94e9e643514 2024-04-02T04:52:10.582 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi122.mwoyoh smithi122 running (9m) 2m ago 9m 15.6M - 17.2.0 e1d6a67b021e 73376d9ee4b8 2024-04-02T04:52:10.582 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi053.legdmv smithi053 *:8443,9283,8765 running (5m) 4s ago 17m 506M - 19.0.0-1979-g6cec793d 78f830190839 c12d516a31bc 2024-04-02T04:52:10.582 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi122.cpwlyk smithi122 *:8443,9283,8765 running (4m) 2m ago 15m 449M - 19.0.0-1979-g6cec793d 78f830190839 9a545c5e7d81 2024-04-02T04:52:10.582 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi053 smithi053 running (4m) 4s ago 17m 50.4M 2048M 19.0.0-1979-g6cec793d 78f830190839 f53a44f19c00 2024-04-02T04:52:10.582 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi122 smithi122 running (3m) 2m ago 14m 34.8M 2048M 19.0.0-1979-g6cec793d 78f830190839 44b200fed3c5 2024-04-02T04:52:10.582 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.0.0.smithi053.mhkdcb smithi053 *:2049 running (9m) 4s ago 9m 296M - 4.0 e1d6a67b021e 07e5da7bd2c5 2024-04-02T04:52:10.582 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.1.0.smithi122.mauvsb smithi122 *:2049 running (9m) 2m ago 9m 81.7M - 4.0 e1d6a67b021e 4f866685005e 2024-04-02T04:52:10.582 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi053 smithi053 *:9100 running (6m) 4s ago 15m 18.4M - 1.3.1 1dbe0e931976 773b71971e67 2024-04-02T04:52:10.582 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi122 smithi122 *:9100 running (5m) 2m ago 14m 15.7M - 1.3.1 1dbe0e931976 475d439c7980 2024-04-02T04:52:10.582 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 smithi053 running (2m) 4s ago 14m 334M 1937M 19.0.0-1979-g6cec793d 78f830190839 490c346c7639 2024-04-02T04:52:10.582 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 smithi053 running (110s) 4s ago 13m 283M 1937M 19.0.0-1979-g6cec793d 78f830190839 e70e40949c2f 2024-04-02T04:52:10.582 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 smithi053 running (62s) 4s ago 13m 200M 1937M 19.0.0-1979-g6cec793d 78f830190839 4f053827e0dd 2024-04-02T04:52:10.583 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 smithi053 running (15s) 4s ago 12m 12.7M 1937M 19.0.0-1979-g6cec793d 78f830190839 5cd6de7a6535 2024-04-02T04:52:10.583 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 smithi122 running (12m) 2m ago 12m 557M 2704M 17.2.0 e1d6a67b021e 46076afcd7bc 2024-04-02T04:52:10.583 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 smithi122 running (12m) 2m ago 12m 389M 2704M 17.2.0 e1d6a67b021e e687a2f6da4d 2024-04-02T04:52:10.583 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 smithi122 running (11m) 2m ago 11m 568M 2704M 17.2.0 e1d6a67b021e 9e371df9a839 2024-04-02T04:52:10.583 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 smithi122 running (11m) 2m ago 11m 480M 2704M 17.2.0 e1d6a67b021e 9b3cc398254d 2024-04-02T04:52:10.583 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus.smithi053 smithi053 *:9095 running (4m) 4s ago 14m 69.6M - 2.33.4 514e6a882f6e 1dd55399da10 2024-04-02T04:52:10.892 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:10 smithi053 ceph-mon[67180]: failed to encode map e112 with expected crc 2024-04-02T04:52:10.892 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:10 smithi053 ceph-mon[67180]: failed to encode map e112 with expected crc 2024-04-02T04:52:10.892 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:10 smithi053 ceph-mon[67180]: osdmap e112: 8 total, 8 up, 8 in 2024-04-02T04:52:10.892 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:10 smithi053 ceph-mon[67180]: failed to encode map e112 with expected crc 2024-04-02T04:52:10.892 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:10 smithi053 ceph-mon[67180]: failed to encode map e112 with expected crc 2024-04-02T04:52:10.892 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:10 smithi053 ceph-mon[67180]: failed to encode map e112 with expected crc 2024-04-02T04:52:10.892 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:10 smithi053 ceph-mon[67180]: from='client.34312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:52:10.892 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:10 smithi053 ceph-mon[67180]: Health check update: Degraded data redundancy: 52/1530 objects degraded (3.399%), 5 pgs degraded (PG_DEGRADED) 2024-04-02T04:52:11.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:10 smithi122 ceph-mon[53211]: failed to encode map e112 with expected crc 2024-04-02T04:52:11.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:10 smithi122 ceph-mon[53211]: failed to encode map e112 with expected crc 2024-04-02T04:52:11.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:10 smithi122 ceph-mon[53211]: osdmap e112: 8 total, 8 up, 8 in 2024-04-02T04:52:11.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:10 smithi122 ceph-mon[53211]: failed to encode map e112 with expected crc 2024-04-02T04:52:11.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:10 smithi122 ceph-mon[53211]: failed to encode map e112 with expected crc 2024-04-02T04:52:11.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:10 smithi122 ceph-mon[53211]: failed to encode map e112 with expected crc 2024-04-02T04:52:11.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:10 smithi122 ceph-mon[53211]: from='client.34312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:52:11.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:10 smithi122 ceph-mon[53211]: Health check update: Degraded data redundancy: 52/1530 objects degraded (3.399%), 5 pgs degraded (PG_DEGRADED) 2024-04-02T04:52:11.271 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:52:11.271 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": { 2024-04-02T04:52:11.271 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:52:11.271 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:52:11.271 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": { 2024-04-02T04:52:11.271 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:52:11.271 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:52:11.271 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": { 2024-04-02T04:52:11.271 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-04-02T04:52:11.271 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 4 2024-04-02T04:52:11.271 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:52:11.271 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": { 2024-04-02T04:52:11.271 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:52:11.271 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:52:11.272 INFO:teuthology.orchestra.run.smithi053.stdout: "overall": { 2024-04-02T04:52:11.272 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-04-02T04:52:11.272 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 8 2024-04-02T04:52:11.272 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-02T04:52:11.272 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:52:11.332 INFO:teuthology.orchestra.run.smithi053.stdout: 5 236281 10.07 MB/sec execute 441 sec latency 37.535 ms 2024-04-02T04:52:11.837 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:52:11.837 INFO:teuthology.orchestra.run.smithi053.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae", 2024-04-02T04:52:11.837 INFO:teuthology.orchestra.run.smithi053.stdout: "in_progress": true, 2024-04-02T04:52:11.837 INFO:teuthology.orchestra.run.smithi053.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-02T04:52:11.837 INFO:teuthology.orchestra.run.smithi053.stdout: "services_complete": [ 2024-04-02T04:52:11.837 INFO:teuthology.orchestra.run.smithi053.stdout: "mon", 2024-04-02T04:52:11.837 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr", 2024-04-02T04:52:11.838 INFO:teuthology.orchestra.run.smithi053.stdout: "crash" 2024-04-02T04:52:11.838 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-04-02T04:52:11.838 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": "10/23 daemons upgraded", 2024-04-02T04:52:11.838 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "Currently upgrading osd daemons", 2024-04-02T04:52:11.838 INFO:teuthology.orchestra.run.smithi053.stdout: "is_paused": false 2024-04-02T04:52:11.838 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:52:11.903 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:11 smithi053 ceph-mon[67180]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:52:11.903 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:11 smithi053 ceph-mon[67180]: pgmap v177: 97 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 94 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 4.0 MiB/s wr, 526 op/s; 8/1497 objects degraded (0.534%); 1.0 MiB/s, 49 objects/s recovering 2024-04-02T04:52:11.903 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:11 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/597399831' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:52:11.903 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:11 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:11.903 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:11 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:52:12.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:11 smithi122 ceph-mon[53211]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:52:12.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:11 smithi122 ceph-mon[53211]: pgmap v177: 97 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 94 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 4.0 MiB/s wr, 526 op/s; 8/1497 objects degraded (0.534%); 1.0 MiB/s, 49 objects/s recovering 2024-04-02T04:52:12.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:11 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/597399831' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:52:12.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:11 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:12.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:11 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:52:12.332 INFO:teuthology.orchestra.run.smithi053.stdout: 5 236595 10.07 MB/sec execute 442 sec latency 40.530 ms 2024-04-02T04:52:12.465 INFO:teuthology.orchestra.run.smithi053.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Degraded data redundancy: 8/1497 objects degraded (0.534%), 1 pg degraded 2024-04-02T04:52:12.465 INFO:teuthology.orchestra.run.smithi053.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-02T04:52:12.466 INFO:teuthology.orchestra.run.smithi053.stdout: daemon grafana.smithi053 on smithi053 is in error state 2024-04-02T04:52:12.466 INFO:teuthology.orchestra.run.smithi053.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 8/1497 objects degraded (0.534%), 1 pg degraded 2024-04-02T04:52:12.466 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.2 is active+recovering+undersized+degraded+remapped, acting [5,6] 2024-04-02T04:52:12.926 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:12 smithi053 ceph-mon[67180]: from='client.34324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:52:12.926 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:12 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/4040345951' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:52:13.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:12 smithi122 ceph-mon[53211]: from='client.34324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:52:13.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:12 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/4040345951' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:52:13.332 INFO:teuthology.orchestra.run.smithi053.stdout: 5 237022 10.09 MB/sec execute 443 sec latency 42.076 ms 2024-04-02T04:52:14.137 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:14 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:14.138 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:14 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:14.138 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:14 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:52:14.138 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:14 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:52:14.138 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:14 smithi122 ceph-mon[53211]: pgmap v178: 97 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 95 active+clean; 336 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.9 MiB/s wr, 514 op/s; 8/1554 objects degraded (0.515%); 770 KiB/s, 36 objects/s recovering 2024-04-02T04:52:14.138 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:14 smithi122 ceph-mon[53211]: pgmap v179: 97 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 95 active+clean; 336 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 3.0 MiB/s wr, 285 op/s; 8/1554 objects degraded (0.515%); 637 KiB/s, 22 objects/s recovering 2024-04-02T04:52:14.138 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:14 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:14.138 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:14 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:14.138 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:14 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:52:14.138 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:14 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:52:14.138 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:14 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:52:14.138 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:14 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:52:14.138 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:14 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-02T04:52:14.139 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:14 smithi122 ceph-mon[53211]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-02T04:52:14.139 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:14 smithi122 ceph-mon[53211]: Upgrade: osd.4 is safe to restart 2024-04-02T04:52:14.204 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:14 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:14.204 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:14 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:14.204 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:14 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:52:14.204 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:14 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:52:14.204 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:14 smithi053 ceph-mon[67180]: pgmap v178: 97 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 95 active+clean; 336 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.9 MiB/s wr, 514 op/s; 8/1554 objects degraded (0.515%); 770 KiB/s, 36 objects/s recovering 2024-04-02T04:52:14.204 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:14 smithi053 ceph-mon[67180]: pgmap v179: 97 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 95 active+clean; 336 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 3.0 MiB/s wr, 285 op/s; 8/1554 objects degraded (0.515%); 637 KiB/s, 22 objects/s recovering 2024-04-02T04:52:14.204 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:14 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:14.205 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:14 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:14.205 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:14 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:52:14.205 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:14 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:52:14.205 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:14 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:52:14.205 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:14 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:52:14.205 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:14 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-02T04:52:14.205 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:14 smithi053 ceph-mon[67180]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-02T04:52:14.205 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:14 smithi053 ceph-mon[67180]: Upgrade: osd.4 is safe to restart 2024-04-02T04:52:14.332 INFO:teuthology.orchestra.run.smithi053.stdout: 5 237434 10.09 MB/sec execute 444 sec latency 42.392 ms 2024-04-02T04:52:15.214 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:15 smithi053 ceph-mon[67180]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8/1554 objects degraded (0.515%), 1 pg degraded) 2024-04-02T04:52:15.333 INFO:teuthology.orchestra.run.smithi053.stdout: 5 237759 10.09 MB/sec execute 445 sec latency 57.124 ms 2024-04-02T04:52:15.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:15 smithi122 ceph-mon[53211]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8/1554 objects degraded (0.515%), 1 pg degraded) 2024-04-02T04:52:16.232 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:16 smithi053 ceph-mon[67180]: pgmap v180: 97 pgs: 97 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.5 MiB/s wr, 280 op/s; 475 KiB/s, 10 objects/s recovering 2024-04-02T04:52:16.232 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:16 smithi053 ceph-mon[67180]: Upgrade: Updating osd.4 2024-04-02T04:52:16.232 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:16 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:16.232 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:16 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-02T04:52:16.232 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:16 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:52:16.232 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:16 smithi053 ceph-mon[67180]: Deploying daemon osd.4 on smithi122 2024-04-02T04:52:16.333 INFO:teuthology.orchestra.run.smithi053.stdout: 5 238074 10.09 MB/sec execute 446 sec latency 41.490 ms 2024-04-02T04:52:16.370 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:16 smithi122 ceph-mon[53211]: pgmap v180: 97 pgs: 97 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.5 MiB/s wr, 280 op/s; 475 KiB/s, 10 objects/s recovering 2024-04-02T04:52:16.370 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:16 smithi122 ceph-mon[53211]: Upgrade: Updating osd.4 2024-04-02T04:52:16.370 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:16 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:16.370 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:16 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-02T04:52:16.370 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:16 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:52:16.370 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:16 smithi122 ceph-mon[53211]: Deploying daemon osd.4 on smithi122 2024-04-02T04:52:17.333 INFO:teuthology.orchestra.run.smithi053.stdout: 5 238350 10.08 MB/sec execute 447 sec latency 60.281 ms 2024-04-02T04:52:18.239 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:18 smithi122 ceph-mon[53211]: pgmap v181: 97 pgs: 97 active+clean; 336 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 3.1 MiB/s wr, 304 op/s; 376 KiB/s, 8 objects/s recovering 2024-04-02T04:52:18.275 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:18 smithi053 ceph-mon[67180]: pgmap v181: 97 pgs: 97 active+clean; 336 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 3.1 MiB/s wr, 304 op/s; 376 KiB/s, 8 objects/s recovering 2024-04-02T04:52:18.333 INFO:teuthology.orchestra.run.smithi053.stdout: 5 238651 10.07 MB/sec execute 448 sec latency 58.403 ms 2024-04-02T04:52:18.895 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:18 smithi122 systemd[1]: Stopping Ceph osd.4 for 48e16742-f0aa-11ee-b647-cb9ed24678a4... 2024-04-02T04:52:19.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:19 smithi053 ceph-mon[67180]: osd.4 marked itself down and dead 2024-04-02T04:52:19.333 INFO:teuthology.orchestra.run.smithi053.stdout: 5 238888 10.06 MB/sec execute 449 sec latency 224.938 ms 2024-04-02T04:52:19.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:19 smithi122 ceph-mon[53211]: osd.4 marked itself down and dead 2024-04-02T04:52:19.395 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:18 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4[31397]: 2024-04-02T04:52:18.922+0000 7fce561f3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T04:52:19.395 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:18 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4[31397]: 2024-04-02T04:52:18.922+0000 7fce561f3700 -1 osd.4 112 *** Got signal Terminated *** 2024-04-02T04:52:19.395 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:18 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4[31397]: 2024-04-02T04:52:18.922+0000 7fce561f3700 -1 osd.4 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T04:52:20.070 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:19 smithi122 podman[57646]: 2024-04-02 04:52:19.745062766 +0000 UTC m=+0.942589417 container died 46076afcd7bcac95b22af8cd9e1e1bf017c77d6493020823ca899950c029f010 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4, io.openshift.tags=base centos centos-stream, architecture=x86_64, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, name=centos-stream, version=8, ceph=True, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-02T04:52:20.333 INFO:teuthology.orchestra.run.smithi053.stdout: 5 238950 10.04 MB/sec execute 450 sec latency 1044.191 ms 2024-04-02T04:52:20.363 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:20 smithi053 ceph-mon[67180]: pgmap v182: 97 pgs: 97 active+clean; 348 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.3 MiB/s wr, 529 op/s; 237 KiB/s, 1 objects/s recovering 2024-04-02T04:52:20.363 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:20 smithi053 ceph-mon[67180]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-02T04:52:20.363 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:20 smithi053 ceph-mon[67180]: failed to encode map e113 with expected crc 2024-04-02T04:52:20.363 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:20 smithi053 ceph-mon[67180]: osdmap e113: 8 total, 7 up, 8 in 2024-04-02T04:52:20.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:20 smithi122 ceph-mon[53211]: pgmap v182: 97 pgs: 97 active+clean; 348 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.3 MiB/s wr, 529 op/s; 237 KiB/s, 1 objects/s recovering 2024-04-02T04:52:20.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:20 smithi122 ceph-mon[53211]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-02T04:52:20.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:20 smithi122 ceph-mon[53211]: failed to encode map e113 with expected crc 2024-04-02T04:52:20.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:20 smithi122 ceph-mon[53211]: osdmap e113: 8 total, 7 up, 8 in 2024-04-02T04:52:20.395 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:20 smithi122 podman[57646]: 2024-04-02 04:52:20.240943172 +0000 UTC m=+1.438469820 container cleanup 46076afcd7bcac95b22af8cd9e1e1bf017c77d6493020823ca899950c029f010 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, maintainer=Guillaume Abrioux , io.openshift.expose-services=, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, com.redhat.component=centos-stream-container, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, name=centos-stream, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, GIT_BRANCH=HEAD) 2024-04-02T04:52:20.395 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:20 smithi122 bash[57646]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4 2024-04-02T04:52:20.865 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:20 smithi122 podman[57665]: 2024-04-02 04:52:20.509318256 +0000 UTC m=+0.768411963 container remove 46076afcd7bcac95b22af8cd9e1e1bf017c77d6493020823ca899950c029f010 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.openshift.expose-services=, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, release=754, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, version=8, ceph=True, com.redhat.component=centos-stream-container, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc.) 2024-04-02T04:52:21.124 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:20 smithi122 podman[57755]: 2024-04-02 04:52:20.765743343 +0000 UTC m=+0.018730748 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:52:21.124 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:20 smithi122 podman[57755]: 2024-04-02 04:52:20.902354055 +0000 UTC m=+0.155341438 container create e7ac0300d034b0b48f504350a788f1827976b3357e4ff651ab58dbec8a287137 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240326, RELEASE=squid-6cec793, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-02T04:52:21.333 INFO:teuthology.orchestra.run.smithi053.stdout: 5 239346 10.04 MB/sec execute 451 sec latency 31.500 ms 2024-04-02T04:52:21.374 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:21 smithi053 ceph-mon[67180]: failed to encode map e114 with expected crc 2024-04-02T04:52:21.374 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:21 smithi053 ceph-mon[67180]: osdmap e114: 8 total, 7 up, 8 in 2024-04-02T04:52:21.374 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:21 smithi053 ceph-mon[67180]: failed to encode map e114 with expected crc 2024-04-02T04:52:21.374 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:21 smithi053 ceph-mon[67180]: failed to encode map e114 with expected crc 2024-04-02T04:52:21.374 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:21 smithi053 ceph-mon[67180]: failed to encode map e114 with expected crc 2024-04-02T04:52:21.374 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:21 smithi053 ceph-mon[67180]: failed to encode map e114 with expected crc 2024-04-02T04:52:21.374 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:21 smithi053 ceph-mon[67180]: failed to encode map e114 with expected crc 2024-04-02T04:52:21.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:21 smithi122 ceph-mon[53211]: failed to encode map e114 with expected crc 2024-04-02T04:52:21.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:21 smithi122 ceph-mon[53211]: osdmap e114: 8 total, 7 up, 8 in 2024-04-02T04:52:21.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:21 smithi122 ceph-mon[53211]: failed to encode map e114 with expected crc 2024-04-02T04:52:21.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:21 smithi122 ceph-mon[53211]: failed to encode map e114 with expected crc 2024-04-02T04:52:21.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:21 smithi122 ceph-mon[53211]: failed to encode map e114 with expected crc 2024-04-02T04:52:21.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:21 smithi122 ceph-mon[53211]: failed to encode map e114 with expected crc 2024-04-02T04:52:21.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:21 smithi122 ceph-mon[53211]: failed to encode map e114 with expected crc 2024-04-02T04:52:21.395 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:21 smithi122 podman[57755]: 2024-04-02 04:52:21.158634228 +0000 UTC m=+0.411621628 container init e7ac0300d034b0b48f504350a788f1827976b3357e4ff651ab58dbec8a287137 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, RELEASE=squid-6cec793, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-02T04:52:21.395 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:21 smithi122 podman[57755]: 2024-04-02 04:52:21.162085365 +0000 UTC m=+0.415072755 container start e7ac0300d034b0b48f504350a788f1827976b3357e4ff651ab58dbec8a287137 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-6cec793, GIT_CLEAN=True) 2024-04-02T04:52:21.395 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:21 smithi122 podman[57755]: 2024-04-02 04:52:21.247194277 +0000 UTC m=+0.500181665 container attach e7ac0300d034b0b48f504350a788f1827976b3357e4ff651ab58dbec8a287137 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-6cec793, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-02T04:52:21.779 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:21 smithi122 podman[57755]: 2024-04-02 04:52:21.473859098 +0000 UTC m=+0.726846485 container died e7ac0300d034b0b48f504350a788f1827976b3357e4ff651ab58dbec8a287137 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-6cec793, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-02T04:52:22.134 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:22 smithi053 ceph-mon[67180]: pgmap v185: 97 pgs: 2 active+undersized, 2 active+undersized+degraded, 8 peering, 10 stale+active+clean, 75 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.6 MiB/s wr, 467 op/s; 19/1593 objects degraded (1.193%); 0 B/s, 1 objects/s recovering 2024-04-02T04:52:22.134 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:22 smithi053 ceph-mon[67180]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-02T04:52:22.134 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:22 smithi053 ceph-mon[67180]: Health check failed: Degraded data redundancy: 19/1593 objects degraded (1.193%), 2 pgs degraded (PG_DEGRADED) 2024-04-02T04:52:22.134 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:22 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:22.333 INFO:teuthology.orchestra.run.smithi053.stdout: 5 239724 10.03 MB/sec execute 452 sec latency 48.078 ms 2024-04-02T04:52:22.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:22 smithi122 ceph-mon[53211]: pgmap v185: 97 pgs: 2 active+undersized, 2 active+undersized+degraded, 8 peering, 10 stale+active+clean, 75 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.6 MiB/s wr, 467 op/s; 19/1593 objects degraded (1.193%); 0 B/s, 1 objects/s recovering 2024-04-02T04:52:22.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:22 smithi122 ceph-mon[53211]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-02T04:52:22.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:22 smithi122 ceph-mon[53211]: Health check failed: Degraded data redundancy: 19/1593 objects degraded (1.193%), 2 pgs degraded (PG_DEGRADED) 2024-04-02T04:52:22.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:22 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:22.395 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:22 smithi122 podman[57782]: 2024-04-02 04:52:22.076605486 +0000 UTC m=+0.596968554 container remove e7ac0300d034b0b48f504350a788f1827976b3357e4ff651ab58dbec8a287137 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-6cec793, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-02T04:52:22.396 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:22 smithi122 systemd[1]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.4.service: Deactivated successfully. 2024-04-02T04:52:22.396 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:22 smithi122 systemd[1]: Stopped Ceph osd.4 for 48e16742-f0aa-11ee-b647-cb9ed24678a4. 2024-04-02T04:52:22.396 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:22 smithi122 systemd[1]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.4.service: Consumed 4min 27.811s CPU time. 2024-04-02T04:52:22.396 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:22 smithi122 systemd[1]: Starting Ceph osd.4 for 48e16742-f0aa-11ee-b647-cb9ed24678a4... 2024-04-02T04:52:22.777 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:22 smithi122 podman[57887]: 2024-04-02 04:52:22.42477793 +0000 UTC m=+0.018700835 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:52:22.777 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:22 smithi122 podman[57887]: 2024-04-02 04:52:22.528422396 +0000 UTC m=+0.122345265 container create 8d5c36ae61e49630b174201bd7775426e7027275dc21e06253f353c43787d2d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4-activate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-6cec793, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-02T04:52:23.145 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:22 smithi122 podman[57887]: 2024-04-02 04:52:22.773870535 +0000 UTC m=+0.367793413 container init 8d5c36ae61e49630b174201bd7775426e7027275dc21e06253f353c43787d2d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4-activate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , RELEASE=squid-6cec793, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-02T04:52:23.145 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:22 smithi122 podman[57887]: 2024-04-02 04:52:22.778418644 +0000 UTC m=+0.372341517 container start 8d5c36ae61e49630b174201bd7775426e7027275dc21e06253f353c43787d2d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4-activate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, RELEASE=squid-6cec793, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux ) 2024-04-02T04:52:23.145 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:22 smithi122 podman[57887]: 2024-04-02 04:52:22.881530121 +0000 UTC m=+0.475453003 container attach 8d5c36ae61e49630b174201bd7775426e7027275dc21e06253f353c43787d2d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4-activate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, RELEASE=squid-6cec793, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T04:52:23.333 INFO:teuthology.orchestra.run.smithi053.stdout: 5 240105 10.04 MB/sec execute 453 sec latency 40.748 ms 2024-04-02T04:52:23.645 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:23 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4-activate[57901]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-02T04:52:23.645 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:23 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4-activate[57901]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-02T04:52:23.645 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:23 smithi122 bash[57887]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-02T04:52:23.645 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:23 smithi122 bash[57887]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-02T04:52:23.645 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:23 smithi122 bash[57887]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-02T04:52:23.645 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:23 smithi122 bash[57887]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-02T04:52:23.646 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:23 smithi122 bash[57887]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-04-02T04:52:23.646 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:23 smithi122 bash[57887]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-02T04:52:23.646 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:23 smithi122 bash[57887]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-02T04:52:23.646 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:23 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4-activate[57901]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-02T04:52:23.646 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:23 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4-activate[57901]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-02T04:52:23.646 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:23 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4-activate[57901]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-04-02T04:52:23.646 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:23 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4-activate[57901]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-02T04:52:23.646 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:23 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4-activate[57901]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-02T04:52:23.646 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:23 smithi122 podman[57887]: 2024-04-02 04:52:23.284498548 +0000 UTC m=+0.878421434 container died 8d5c36ae61e49630b174201bd7775426e7027275dc21e06253f353c43787d2d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4-activate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-6cec793, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T04:52:24.189 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:24 smithi053 ceph-mon[67180]: pgmap v186: 97 pgs: 8 active+undersized, 20 active+undersized+degraded, 8 peering, 3 stale+active+clean, 58 active+clean; 330 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.1 MiB/s wr, 599 op/s; 180/1560 objects degraded (11.538%) 2024-04-02T04:52:24.282 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:24 smithi122 ceph-mon[53211]: pgmap v186: 97 pgs: 8 active+undersized, 20 active+undersized+degraded, 8 peering, 3 stale+active+clean, 58 active+clean; 330 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.1 MiB/s wr, 599 op/s; 180/1560 objects degraded (11.538%) 2024-04-02T04:52:24.282 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:24 smithi122 podman[58024]: 2024-04-02 04:52:24.011721226 +0000 UTC m=+0.716480707 container remove 8d5c36ae61e49630b174201bd7775426e7027275dc21e06253f353c43787d2d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4-activate, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-6cec793, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD) 2024-04-02T04:52:24.282 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:24 smithi122 podman[58078]: 2024-04-02 04:52:24.153413495 +0000 UTC m=+0.024321968 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:52:24.333 INFO:teuthology.orchestra.run.smithi053.stdout: 5 240373 10.04 MB/sec execute 454 sec latency 71.874 ms 2024-04-02T04:52:24.543 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:24 smithi122 podman[58078]: 2024-04-02 04:52:24.279463612 +0000 UTC m=+0.150372053 container create 40d6ab1aac4b114be6c69ce2ec01ed0e4b5ad9d84c2f66cbb482ec407c213a01 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-6cec793) 2024-04-02T04:52:24.543 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:24 smithi122 podman[58078]: 2024-04-02 04:52:24.515803483 +0000 UTC m=+0.386711933 container init 40d6ab1aac4b114be6c69ce2ec01ed0e4b5ad9d84c2f66cbb482ec407c213a01 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-6cec793, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-02T04:52:24.543 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:24 smithi122 podman[58078]: 2024-04-02 04:52:24.519355069 +0000 UTC m=+0.390263513 container start 40d6ab1aac4b114be6c69ce2ec01ed0e4b5ad9d84c2f66cbb482ec407c213a01 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-6cec793, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326) 2024-04-02T04:52:24.543 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:24 smithi122 ceph-osd[58096]: -- 172.21.15.122:0/574693639 <== mon.1 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5570e6559680 con 0x5570e6523000 2024-04-02T04:52:24.894 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:24 smithi122 bash[58078]: 40d6ab1aac4b114be6c69ce2ec01ed0e4b5ad9d84c2f66cbb482ec407c213a01 2024-04-02T04:52:24.895 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:24 smithi122 systemd[1]: Started Ceph osd.4 for 48e16742-f0aa-11ee-b647-cb9ed24678a4. 2024-04-02T04:52:25.333 INFO:teuthology.orchestra.run.smithi053.stdout: 5 240684 10.04 MB/sec execute 455 sec latency 72.491 ms 2024-04-02T04:52:25.395 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:25 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4[58092]: 2024-04-02T04:52:25.062+0000 7f36985bf7c0 -1 Falling back to public interface 2024-04-02T04:52:25.739 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:25.739 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:25.739 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:52:25.739 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:25 smithi053 ceph-mon[67180]: pgmap v187: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 331 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.6 MiB/s wr, 573 op/s; 252/1581 objects degraded (15.939%) 2024-04-02T04:52:25.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:25.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:25.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:52:25.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:25 smithi122 ceph-mon[53211]: pgmap v187: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 331 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.6 MiB/s wr, 573 op/s; 252/1581 objects degraded (15.939%) 2024-04-02T04:52:26.333 INFO:teuthology.orchestra.run.smithi053.stdout: 5 240972 10.03 MB/sec execute 456 sec latency 58.989 ms 2024-04-02T04:52:26.759 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:26 smithi053 ceph-mon[67180]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-02T04:52:26.759 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:26 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:26.759 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:26 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:52:27.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:26 smithi122 ceph-mon[53211]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-02T04:52:27.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:26 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:27.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:26 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:52:27.333 INFO:teuthology.orchestra.run.smithi053.stdout: 5 241208 10.03 MB/sec execute 457 sec latency 82.034 ms 2024-04-02T04:52:27.781 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:27 smithi053 ceph-mon[67180]: pgmap v188: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 331 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 2.4 MiB/s wr, 253 op/s; 252/1581 objects degraded (15.939%) 2024-04-02T04:52:27.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:27 smithi122 ceph-mon[53211]: pgmap v188: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 331 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 2.4 MiB/s wr, 253 op/s; 252/1581 objects degraded (15.939%) 2024-04-02T04:52:28.333 INFO:teuthology.orchestra.run.smithi053.stdout: 5 241503 10.03 MB/sec execute 458 sec latency 67.420 ms 2024-04-02T04:52:29.333 INFO:teuthology.orchestra.run.smithi053.stdout: 5 241749 10.02 MB/sec execute 459 sec latency 48.069 ms 2024-04-02T04:52:29.568 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:29 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:29.568 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:29 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:29.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:29 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:29.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:29 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:30.333 INFO:teuthology.orchestra.run.smithi053.stdout: 5 242061 10.01 MB/sec execute 460 sec latency 47.906 ms 2024-04-02T04:52:30.583 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:30 smithi053 ceph-mon[67180]: pgmap v189: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.0 MiB/s wr, 390 op/s; 260/1638 objects degraded (15.873%) 2024-04-02T04:52:30.583 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:30 smithi053 ceph-mon[67180]: Health check update: Degraded data redundancy: 260/1638 objects degraded (15.873%), 28 pgs degraded (PG_DEGRADED) 2024-04-02T04:52:30.584 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:30 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:30.584 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:30 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:30.765 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:30 smithi122 ceph-mon[53211]: pgmap v189: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.0 MiB/s wr, 390 op/s; 260/1638 objects degraded (15.873%) 2024-04-02T04:52:30.765 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:30 smithi122 ceph-mon[53211]: Health check update: Degraded data redundancy: 260/1638 objects degraded (15.873%), 28 pgs degraded (PG_DEGRADED) 2024-04-02T04:52:30.765 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:30 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:30.765 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:30 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:31.333 INFO:teuthology.orchestra.run.smithi053.stdout: 5 242417 10.01 MB/sec execute 461 sec latency 45.581 ms 2024-04-02T04:52:31.843 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:31 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4[58092]: 2024-04-02T04:52:31.554+0000 7f36985bf7c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-04-02T04:52:32.333 INFO:teuthology.orchestra.run.smithi053.stdout: 5 242803 10.00 MB/sec execute 462 sec latency 39.400 ms 2024-04-02T04:52:32.632 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:32 smithi053 ceph-mon[67180]: pgmap v190: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 3.8 MiB/s wr, 358 op/s; 265/1659 objects degraded (15.973%) 2024-04-02T04:52:32.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:32 smithi122 ceph-mon[53211]: pgmap v190: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 3.8 MiB/s wr, 358 op/s; 265/1659 objects degraded (15.973%) 2024-04-02T04:52:33.333 INFO:teuthology.orchestra.run.smithi053.stdout: 5 243211 10.00 MB/sec execute 463 sec latency 49.280 ms 2024-04-02T04:52:33.394 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:33 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4[58092]: 2024-04-02T04:52:33.158+0000 7f36985bf7c0 -1 osd.4 112 log_to_monitors true 2024-04-02T04:52:33.663 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:33 smithi053 ceph-mon[67180]: from='osd.4 [v2:172.21.15.122:6800/3182580439,v1:172.21.15.122:6801/3182580439]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-02T04:52:33.663 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:33 smithi053 ceph-mon[67180]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-02T04:52:33.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:33 smithi122 ceph-mon[53211]: from='osd.4 [v2:172.21.15.122:6800/3182580439,v1:172.21.15.122:6801/3182580439]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-02T04:52:33.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:33 smithi122 ceph-mon[53211]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-02T04:52:34.333 INFO:teuthology.orchestra.run.smithi053.stdout: 5 243513 9.99 MB/sec execute 464 sec latency 166.842 ms 2024-04-02T04:52:34.394 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:52:33 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4[58092]: 2024-04-02T04:52:33.985+0000 7f3685a0f700 -1 osd.4 112 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T04:52:34.668 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:34 smithi053 ceph-mon[67180]: pgmap v191: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 340 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.9 MiB/s wr, 504 op/s; 261/1665 objects degraded (15.676%) 2024-04-02T04:52:34.668 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:34 smithi053 ceph-mon[67180]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-02T04:52:34.669 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:34 smithi053 ceph-mon[67180]: failed to encode map e115 with expected crc 2024-04-02T04:52:34.669 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:34 smithi053 ceph-mon[67180]: osdmap e115: 8 total, 7 up, 8 in 2024-04-02T04:52:34.669 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:34 smithi053 ceph-mon[67180]: failed to encode map e115 with expected crc 2024-04-02T04:52:34.669 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:34 smithi053 ceph-mon[67180]: from='osd.4 [v2:172.21.15.122:6800/3182580439,v1:172.21.15.122:6801/3182580439]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-02T04:52:34.669 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:34 smithi053 ceph-mon[67180]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-02T04:52:34.669 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:34 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:34.669 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:34 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:34.669 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:34 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:52:34.669 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:34 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:52:34.669 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:34 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:34.669 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:34 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:34.669 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:34 smithi053 ceph-mon[67180]: from='osd.4 ' entity='osd.4' 2024-04-02T04:52:34.669 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:34 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:52:34.669 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:34 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:52:34.669 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:34 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:52:34.669 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:34 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:52:34.670 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:34 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-02T04:52:34.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:34 smithi122 ceph-mon[53211]: pgmap v191: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 340 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.9 MiB/s wr, 504 op/s; 261/1665 objects degraded (15.676%) 2024-04-02T04:52:34.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:34 smithi122 ceph-mon[53211]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-02T04:52:34.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:34 smithi122 ceph-mon[53211]: failed to encode map e115 with expected crc 2024-04-02T04:52:34.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:34 smithi122 ceph-mon[53211]: osdmap e115: 8 total, 7 up, 8 in 2024-04-02T04:52:34.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:34 smithi122 ceph-mon[53211]: failed to encode map e115 with expected crc 2024-04-02T04:52:34.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:34 smithi122 ceph-mon[53211]: from='osd.4 [v2:172.21.15.122:6800/3182580439,v1:172.21.15.122:6801/3182580439]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-02T04:52:34.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:34 smithi122 ceph-mon[53211]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-02T04:52:34.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:34 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:34.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:34 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:34.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:34 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:52:34.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:34 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:52:34.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:34 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:34.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:34 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:34.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:34 smithi122 ceph-mon[53211]: from='osd.4 ' entity='osd.4' 2024-04-02T04:52:34.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:34 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:52:34.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:34 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:52:34.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:34 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:52:34.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:34 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:52:34.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:34 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-02T04:52:35.333 INFO:teuthology.orchestra.run.smithi053.stdout: 5 243770 9.99 MB/sec execute 465 sec latency 341.859 ms 2024-04-02T04:52:35.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:35 smithi053 ceph-mon[67180]: pgmap v193: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 337 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 4.9 MiB/s wr, 495 op/s; 256/1599 objects degraded (16.010%) 2024-04-02T04:52:35.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:35 smithi053 ceph-mon[67180]: pgmap v194: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 337 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.3 MiB/s wr, 634 op/s; 256/1599 objects degraded (16.010%) 2024-04-02T04:52:35.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:35 smithi053 ceph-mon[67180]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-02T04:52:35.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:35 smithi053 ceph-mon[67180]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-04-02T04:52:35.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:35 smithi053 ceph-mon[67180]: Health check update: Degraded data redundancy: 256/1599 objects degraded (16.010%), 28 pgs degraded (PG_DEGRADED) 2024-04-02T04:52:35.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:35 smithi053 ceph-mon[67180]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-02T04:52:35.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:35 smithi053 ceph-mon[67180]: failed to encode map e116 with expected crc 2024-04-02T04:52:35.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:35 smithi053 ceph-mon[67180]: osd.4 [v2:172.21.15.122:6800/3182580439,v1:172.21.15.122:6801/3182580439] boot 2024-04-02T04:52:35.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:35 smithi053 ceph-mon[67180]: osdmap e116: 8 total, 8 up, 8 in 2024-04-02T04:52:35.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:35 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:52:35.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:35 smithi122 ceph-mon[53211]: pgmap v193: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 337 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 4.9 MiB/s wr, 495 op/s; 256/1599 objects degraded (16.010%) 2024-04-02T04:52:35.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:35 smithi122 ceph-mon[53211]: pgmap v194: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 337 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.3 MiB/s wr, 634 op/s; 256/1599 objects degraded (16.010%) 2024-04-02T04:52:35.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:35 smithi122 ceph-mon[53211]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-02T04:52:35.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:35 smithi122 ceph-mon[53211]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-04-02T04:52:35.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:35 smithi122 ceph-mon[53211]: Health check update: Degraded data redundancy: 256/1599 objects degraded (16.010%), 28 pgs degraded (PG_DEGRADED) 2024-04-02T04:52:35.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:35 smithi122 ceph-mon[53211]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-02T04:52:35.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:35 smithi122 ceph-mon[53211]: failed to encode map e116 with expected crc 2024-04-02T04:52:35.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:35 smithi122 ceph-mon[53211]: osd.4 [v2:172.21.15.122:6800/3182580439,v1:172.21.15.122:6801/3182580439] boot 2024-04-02T04:52:35.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:35 smithi122 ceph-mon[53211]: osdmap e116: 8 total, 8 up, 8 in 2024-04-02T04:52:35.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:35 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T04:52:36.333 INFO:teuthology.orchestra.run.smithi053.stdout: 5 243979 9.99 MB/sec execute 466 sec latency 1194.362 ms 2024-04-02T04:52:36.989 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:36 smithi053 ceph-mon[67180]: osdmap e117: 8 total, 8 up, 8 in 2024-04-02T04:52:36.989 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:36 smithi053 ceph-mon[67180]: failed to encode map e117 with expected crc 2024-04-02T04:52:36.989 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:36 smithi053 ceph-mon[67180]: failed to encode map e117 with expected crc 2024-04-02T04:52:36.989 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:36 smithi053 ceph-mon[67180]: failed to encode map e117 with expected crc 2024-04-02T04:52:36.989 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:36 smithi053 ceph-mon[67180]: pgmap v197: 97 pgs: 2 peering, 12 active+undersized, 27 active+undersized+degraded, 56 active+clean; 330 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 717 B/s rd, 2.4 MiB/s wr, 259 op/s; 240/1551 objects degraded (15.474%) 2024-04-02T04:52:37.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:36 smithi122 ceph-mon[53211]: osdmap e117: 8 total, 8 up, 8 in 2024-04-02T04:52:37.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:36 smithi122 ceph-mon[53211]: failed to encode map e117 with expected crc 2024-04-02T04:52:37.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:36 smithi122 ceph-mon[53211]: failed to encode map e117 with expected crc 2024-04-02T04:52:37.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:36 smithi122 ceph-mon[53211]: failed to encode map e117 with expected crc 2024-04-02T04:52:37.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:36 smithi122 ceph-mon[53211]: pgmap v197: 97 pgs: 2 peering, 12 active+undersized, 27 active+undersized+degraded, 56 active+clean; 330 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 717 B/s rd, 2.4 MiB/s wr, 259 op/s; 240/1551 objects degraded (15.474%) 2024-04-02T04:52:37.334 INFO:teuthology.orchestra.run.smithi053.stdout: 5 244171 9.98 MB/sec execute 467 sec latency 1640.963 ms 2024-04-02T04:52:37.758 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:37 smithi053 ceph-mon[67180]: failed to encode map e118 with expected crc 2024-04-02T04:52:37.758 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:37 smithi053 ceph-mon[67180]: osdmap e118: 8 total, 8 up, 8 in 2024-04-02T04:52:37.758 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:37 smithi053 ceph-mon[67180]: failed to encode map e118 with expected crc 2024-04-02T04:52:37.758 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:37 smithi053 ceph-mon[67180]: failed to encode map e118 with expected crc 2024-04-02T04:52:37.758 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:37 smithi053 ceph-mon[67180]: failed to encode map e118 with expected crc 2024-04-02T04:52:37.758 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:37 smithi053 ceph-mon[67180]: failed to encode map e118 with expected crc 2024-04-02T04:52:37.758 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:37 smithi053 ceph-mon[67180]: failed to encode map e118 with expected crc 2024-04-02T04:52:38.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:37 smithi122 ceph-mon[53211]: failed to encode map e118 with expected crc 2024-04-02T04:52:38.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:37 smithi122 ceph-mon[53211]: osdmap e118: 8 total, 8 up, 8 in 2024-04-02T04:52:38.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:37 smithi122 ceph-mon[53211]: failed to encode map e118 with expected crc 2024-04-02T04:52:38.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:37 smithi122 ceph-mon[53211]: failed to encode map e118 with expected crc 2024-04-02T04:52:38.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:37 smithi122 ceph-mon[53211]: failed to encode map e118 with expected crc 2024-04-02T04:52:38.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:37 smithi122 ceph-mon[53211]: failed to encode map e118 with expected crc 2024-04-02T04:52:38.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:37 smithi122 ceph-mon[53211]: failed to encode map e118 with expected crc 2024-04-02T04:52:38.334 INFO:teuthology.orchestra.run.smithi053.stdout: 5 244577 9.99 MB/sec execute 468 sec latency 521.338 ms 2024-04-02T04:52:38.780 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:38 smithi053 ceph-mon[67180]: failed to encode map e119 with expected crc 2024-04-02T04:52:38.780 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:38 smithi053 ceph-mon[67180]: osdmap e119: 8 total, 8 up, 8 in 2024-04-02T04:52:38.780 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:38 smithi053 ceph-mon[67180]: failed to encode map e119 with expected crc 2024-04-02T04:52:38.780 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:38 smithi053 ceph-mon[67180]: failed to encode map e119 with expected crc 2024-04-02T04:52:38.780 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:38 smithi053 ceph-mon[67180]: failed to encode map e119 with expected crc 2024-04-02T04:52:38.780 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:38 smithi053 ceph-mon[67180]: failed to encode map e119 with expected crc 2024-04-02T04:52:38.780 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:38 smithi053 ceph-mon[67180]: pgmap v200: 97 pgs: 2 peering, 11 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+undersized+remapped, 10 active+undersized+degraded, 68 active+clean; 329 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.2 MiB/s wr, 358 op/s; 118/1581 objects degraded (7.464%); 2.1 MiB/s, 59 keys/s, 35 objects/s recovering 2024-04-02T04:52:39.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:38 smithi122 ceph-mon[53211]: failed to encode map e119 with expected crc 2024-04-02T04:52:39.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:38 smithi122 ceph-mon[53211]: osdmap e119: 8 total, 8 up, 8 in 2024-04-02T04:52:39.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:38 smithi122 ceph-mon[53211]: failed to encode map e119 with expected crc 2024-04-02T04:52:39.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:38 smithi122 ceph-mon[53211]: failed to encode map e119 with expected crc 2024-04-02T04:52:39.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:38 smithi122 ceph-mon[53211]: failed to encode map e119 with expected crc 2024-04-02T04:52:39.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:38 smithi122 ceph-mon[53211]: failed to encode map e119 with expected crc 2024-04-02T04:52:39.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:38 smithi122 ceph-mon[53211]: pgmap v200: 97 pgs: 2 peering, 11 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+undersized+remapped, 10 active+undersized+degraded, 68 active+clean; 329 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.2 MiB/s wr, 358 op/s; 118/1581 objects degraded (7.464%); 2.1 MiB/s, 59 keys/s, 35 objects/s recovering 2024-04-02T04:52:39.334 INFO:teuthology.orchestra.run.smithi053.stdout: 5 244763 9.98 MB/sec execute 469 sec latency 987.325 ms 2024-04-02T04:52:40.027 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:39 smithi053 ceph-mon[67180]: failed to encode map e120 with expected crc 2024-04-02T04:52:40.027 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:39 smithi053 ceph-mon[67180]: osdmap e120: 8 total, 8 up, 8 in 2024-04-02T04:52:40.027 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:39 smithi053 ceph-mon[67180]: failed to encode map e120 with expected crc 2024-04-02T04:52:40.027 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:39 smithi053 ceph-mon[67180]: failed to encode map e120 with expected crc 2024-04-02T04:52:40.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:39 smithi122 ceph-mon[53211]: failed to encode map e120 with expected crc 2024-04-02T04:52:40.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:39 smithi122 ceph-mon[53211]: osdmap e120: 8 total, 8 up, 8 in 2024-04-02T04:52:40.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:39 smithi122 ceph-mon[53211]: failed to encode map e120 with expected crc 2024-04-02T04:52:40.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:39 smithi122 ceph-mon[53211]: failed to encode map e120 with expected crc 2024-04-02T04:52:40.333 INFO:teuthology.orchestra.run.smithi053.stdout: 5 244869 9.96 MB/sec execute 470 sec latency 977.465 ms 2024-04-02T04:52:41.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:40 smithi053 ceph-mon[67180]: osdmap e121: 8 total, 8 up, 8 in 2024-04-02T04:52:41.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:40 smithi053 ceph-mon[67180]: failed to encode map e121 with expected crc 2024-04-02T04:52:41.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:40 smithi053 ceph-mon[67180]: failed to encode map e121 with expected crc 2024-04-02T04:52:41.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:40 smithi053 ceph-mon[67180]: pgmap v203: 97 pgs: 4 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+remapped, 88 active+clean; 332 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.3 MiB/s wr, 275 op/s; 25/1644 objects degraded (1.521%); 2.3 MiB/s, 59 keys/s, 65 objects/s recovering 2024-04-02T04:52:41.046 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:40 smithi053 ceph-mon[67180]: Health check update: Degraded data redundancy: 118/1581 objects degraded (7.464%), 12 pgs degraded (PG_DEGRADED) 2024-04-02T04:52:41.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:40 smithi122 ceph-mon[53211]: osdmap e121: 8 total, 8 up, 8 in 2024-04-02T04:52:41.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:40 smithi122 ceph-mon[53211]: failed to encode map e121 with expected crc 2024-04-02T04:52:41.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:40 smithi122 ceph-mon[53211]: failed to encode map e121 with expected crc 2024-04-02T04:52:41.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:40 smithi122 ceph-mon[53211]: pgmap v203: 97 pgs: 4 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+remapped, 88 active+clean; 332 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.3 MiB/s wr, 275 op/s; 25/1644 objects degraded (1.521%); 2.3 MiB/s, 59 keys/s, 65 objects/s recovering 2024-04-02T04:52:41.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:40 smithi122 ceph-mon[53211]: Health check update: Degraded data redundancy: 118/1581 objects degraded (7.464%), 12 pgs degraded (PG_DEGRADED) 2024-04-02T04:52:41.334 INFO:teuthology.orchestra.run.smithi053.stdout: 5 245081 9.95 MB/sec execute 471 sec latency 35.959 ms 2024-04-02T04:52:42.334 INFO:teuthology.orchestra.run.smithi053.stdout: 5 245397 9.95 MB/sec execute 472 sec latency 61.471 ms 2024-04-02T04:52:42.839 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:42 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:42.839 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:42 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:52:42.839 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:42 smithi053 ceph-mon[67180]: pgmap v204: 97 pgs: 2 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+remapped, 90 active+clean; 336 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 4.3 MiB/s wr, 417 op/s; 25/1647 objects degraded (1.518%); 1.8 MiB/s, 46 keys/s, 50 objects/s recovering 2024-04-02T04:52:42.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:42 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:42.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:42 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:52:42.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:42 smithi122 ceph-mon[53211]: pgmap v204: 97 pgs: 2 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+remapped, 90 active+clean; 336 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 4.3 MiB/s wr, 417 op/s; 25/1647 objects degraded (1.518%); 1.8 MiB/s, 46 keys/s, 50 objects/s recovering 2024-04-02T04:52:42.906 INFO:teuthology.orchestra.run.smithi053.stdout:true 2024-04-02T04:52:43.334 INFO:teuthology.orchestra.run.smithi053.stdout: 5 245683 9.93 MB/sec execute 473 sec latency 53.200 ms 2024-04-02T04:52:43.729 INFO:teuthology.orchestra.run.smithi053.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-02T04:52:43.729 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager.smithi053 smithi053 *:9093,9094 running (7m) 37s ago 16m 23.6M - 0.23.0 ba2b418f427c bf50c1cd2b3d 2024-04-02T04:52:43.730 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi053 smithi053 running (3m) 37s ago 16m 6660k - 19.0.0-1979-g6cec793d 78f830190839 4ce8915e0157 2024-04-02T04:52:43.730 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi122 smithi122 running (3m) 15s ago 15m 6660k - 19.0.0-1979-g6cec793d 78f830190839 e2b5369e6fdf 2024-04-02T04:52:43.730 INFO:teuthology.orchestra.run.smithi053.stdout:grafana.smithi053 smithi053 *:3000 error 37s ago 16m - - 2024-04-02T04:52:43.730 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi053.nfrfot smithi053 running (10m) 37s ago 10m 37.4M - 17.2.0 e1d6a67b021e c94e9e643514 2024-04-02T04:52:43.730 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi122.mwoyoh smithi122 running (10m) 15s ago 10m 16.1M - 17.2.0 e1d6a67b021e 73376d9ee4b8 2024-04-02T04:52:43.730 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi053.legdmv smithi053 *:8443,9283,8765 running (6m) 37s ago 17m 506M - 19.0.0-1979-g6cec793d 78f830190839 c12d516a31bc 2024-04-02T04:52:43.730 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi122.cpwlyk smithi122 *:8443,9283,8765 running (5m) 15s ago 15m 451M - 19.0.0-1979-g6cec793d 78f830190839 9a545c5e7d81 2024-04-02T04:52:43.730 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi053 smithi053 running (4m) 37s ago 17m 50.4M 2048M 19.0.0-1979-g6cec793d 78f830190839 f53a44f19c00 2024-04-02T04:52:43.730 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi122 smithi122 running (4m) 15s ago 15m 40.4M 2048M 19.0.0-1979-g6cec793d 78f830190839 44b200fed3c5 2024-04-02T04:52:43.730 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.0.0.smithi053.mhkdcb smithi053 *:2049 running (10m) 37s ago 10m 296M - 4.0 e1d6a67b021e 07e5da7bd2c5 2024-04-02T04:52:43.730 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.1.0.smithi122.mauvsb smithi122 *:2049 running (10m) 15s ago 10m 84.2M - 4.0 e1d6a67b021e 4f866685005e 2024-04-02T04:52:43.730 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi053 smithi053 *:9100 running (6m) 37s ago 15m 18.4M - 1.3.1 1dbe0e931976 773b71971e67 2024-04-02T04:52:43.730 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi122 smithi122 *:9100 running (6m) 15s ago 15m 16.6M - 1.3.1 1dbe0e931976 475d439c7980 2024-04-02T04:52:43.730 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 smithi053 running (2m) 37s ago 14m 334M 1937M 19.0.0-1979-g6cec793d 78f830190839 490c346c7639 2024-04-02T04:52:43.731 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 smithi053 running (2m) 37s ago 14m 283M 1937M 19.0.0-1979-g6cec793d 78f830190839 e70e40949c2f 2024-04-02T04:52:43.731 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 smithi053 running (96s) 37s ago 13m 200M 1937M 19.0.0-1979-g6cec793d 78f830190839 4f053827e0dd 2024-04-02T04:52:43.731 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 smithi053 running (48s) 37s ago 13m 12.7M 1937M 19.0.0-1979-g6cec793d 78f830190839 5cd6de7a6535 2024-04-02T04:52:43.731 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 smithi122 running (19s) 15s ago 13m 12.6M 2704M 19.0.0-1979-g6cec793d 78f830190839 40d6ab1aac4b 2024-04-02T04:52:43.731 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 smithi122 running (12m) 15s ago 12m 560M 2704M 17.2.0 e1d6a67b021e e687a2f6da4d 2024-04-02T04:52:43.731 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 smithi122 running (12m) 15s ago 12m 484M 2704M 17.2.0 e1d6a67b021e 9e371df9a839 2024-04-02T04:52:43.731 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 smithi122 running (11m) 15s ago 11m 749M 2704M 17.2.0 e1d6a67b021e 9b3cc398254d 2024-04-02T04:52:43.731 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus.smithi053 smithi053 *:9095 running (5m) 37s ago 15m 69.6M - 2.33.4 514e6a882f6e 1dd55399da10 2024-04-02T04:52:43.862 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:43 smithi053 ceph-mon[67180]: from='client.34332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:52:43.862 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:43 smithi053 ceph-mon[67180]: from='client.34336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:52:43.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:43 smithi122 ceph-mon[53211]: from='client.34332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:52:43.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:43 smithi122 ceph-mon[53211]: from='client.34336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:52:44.229 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:52:44.229 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": { 2024-04-02T04:52:44.229 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:52:44.229 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:52:44.229 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": { 2024-04-02T04:52:44.230 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:52:44.230 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:52:44.230 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": { 2024-04-02T04:52:44.230 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-04-02T04:52:44.230 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 5 2024-04-02T04:52:44.230 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:52:44.230 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": { 2024-04-02T04:52:44.230 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:52:44.230 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:52:44.230 INFO:teuthology.orchestra.run.smithi053.stdout: "overall": { 2024-04-02T04:52:44.230 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-04-02T04:52:44.230 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 9 2024-04-02T04:52:44.230 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-02T04:52:44.230 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:52:44.334 INFO:teuthology.orchestra.run.smithi053.stdout: 5 245916 9.93 MB/sec execute 474 sec latency 59.662 ms 2024-04-02T04:52:44.633 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:44 smithi053 ceph-mon[67180]: from='client.34340 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:52:44.633 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:44 smithi053 ceph-mon[67180]: pgmap v205: 97 pgs: 1 peering, 96 active+clean; 334 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 3.2 MiB/s wr, 288 op/s; 103 KiB/s, 24 objects/s recovering 2024-04-02T04:52:44.633 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:44 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/429920487' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:52:44.644 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:52:44.644 INFO:teuthology.orchestra.run.smithi053.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae", 2024-04-02T04:52:44.644 INFO:teuthology.orchestra.run.smithi053.stdout: "in_progress": true, 2024-04-02T04:52:44.644 INFO:teuthology.orchestra.run.smithi053.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-02T04:52:44.644 INFO:teuthology.orchestra.run.smithi053.stdout: "services_complete": [ 2024-04-02T04:52:44.644 INFO:teuthology.orchestra.run.smithi053.stdout: "mon", 2024-04-02T04:52:44.644 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr", 2024-04-02T04:52:44.644 INFO:teuthology.orchestra.run.smithi053.stdout: "crash" 2024-04-02T04:52:44.644 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-04-02T04:52:44.645 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": "11/23 daemons upgraded", 2024-04-02T04:52:44.645 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "Currently upgrading osd daemons", 2024-04-02T04:52:44.645 INFO:teuthology.orchestra.run.smithi053.stdout: "is_paused": false 2024-04-02T04:52:44.645 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:52:44.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:44 smithi122 ceph-mon[53211]: from='client.34340 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:52:44.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:44 smithi122 ceph-mon[53211]: pgmap v205: 97 pgs: 1 peering, 96 active+clean; 334 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 3.2 MiB/s wr, 288 op/s; 103 KiB/s, 24 objects/s recovering 2024-04-02T04:52:44.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:44 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/429920487' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:52:45.126 INFO:teuthology.orchestra.run.smithi053.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-02T04:52:45.126 INFO:teuthology.orchestra.run.smithi053.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-02T04:52:45.126 INFO:teuthology.orchestra.run.smithi053.stdout: daemon grafana.smithi053 on smithi053 is in error state 2024-04-02T04:52:45.334 INFO:teuthology.orchestra.run.smithi053.stdout: 5 246145 9.93 MB/sec execute 475 sec latency 55.586 ms 2024-04-02T04:52:45.679 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:45 smithi053 ceph-mon[67180]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 25/1647 objects degraded (1.518%), 2 pgs degraded) 2024-04-02T04:52:45.680 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:45 smithi053 ceph-mon[67180]: from='client.34348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:52:45.680 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:45 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/3226895407' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:52:45.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:45 smithi122 ceph-mon[53211]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 25/1647 objects degraded (1.518%), 2 pgs degraded) 2024-04-02T04:52:45.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:45 smithi122 ceph-mon[53211]: from='client.34348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:52:45.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:45 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/3226895407' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:52:46.334 INFO:teuthology.orchestra.run.smithi053.stdout: 5 246439 9.92 MB/sec execute 476 sec latency 73.872 ms 2024-04-02T04:52:46.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:46 smithi053 ceph-mon[67180]: pgmap v206: 97 pgs: 97 active+clean; 335 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.2 MiB/s wr, 309 op/s; 79 KiB/s, 18 objects/s recovering 2024-04-02T04:52:46.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:46 smithi122 ceph-mon[53211]: pgmap v206: 97 pgs: 97 active+clean; 335 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.2 MiB/s wr, 309 op/s; 79 KiB/s, 18 objects/s recovering 2024-04-02T04:52:47.334 INFO:teuthology.orchestra.run.smithi053.stdout: 5 246782 9.91 MB/sec execute 477 sec latency 69.559 ms 2024-04-02T04:52:48.334 INFO:teuthology.orchestra.run.smithi053.stdout: 5 247083 9.91 MB/sec execute 478 sec latency 49.569 ms 2024-04-02T04:52:49.007 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:48 smithi053 ceph-mon[67180]: pgmap v207: 97 pgs: 97 active+clean; 330 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 3.6 MiB/s wr, 427 op/s; 47 KiB/s, 8 objects/s recovering 2024-04-02T04:52:49.334 INFO:teuthology.orchestra.run.smithi053.stdout: 5 247308 9.90 MB/sec execute 479 sec latency 44.726 ms 2024-04-02T04:52:49.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:48 smithi122 ceph-mon[53211]: pgmap v207: 97 pgs: 97 active+clean; 330 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 3.6 MiB/s wr, 427 op/s; 47 KiB/s, 8 objects/s recovering 2024-04-02T04:52:50.008 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:49 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-02T04:52:50.008 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:49 smithi122 ceph-mon[53211]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-02T04:52:50.008 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:49 smithi122 ceph-mon[53211]: Upgrade: osd.5 is safe to restart 2024-04-02T04:52:50.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:49 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-02T04:52:50.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:49 smithi053 ceph-mon[67180]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-02T04:52:50.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:49 smithi053 ceph-mon[67180]: Upgrade: osd.5 is safe to restart 2024-04-02T04:52:50.334 INFO:teuthology.orchestra.run.smithi053.stdout: 5 247628 9.91 MB/sec execute 480 sec latency 51.618 ms 2024-04-02T04:52:51.128 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:50 smithi053 ceph-mon[67180]: pgmap v208: 97 pgs: 97 active+clean; 331 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 3.8 MiB/s wr, 412 op/s; 0 B/s, 3 objects/s recovering 2024-04-02T04:52:51.128 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:50 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:51.128 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:50 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-02T04:52:51.128 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:50 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:52:51.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:50 smithi122 ceph-mon[53211]: pgmap v208: 97 pgs: 97 active+clean; 331 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 3.8 MiB/s wr, 412 op/s; 0 B/s, 3 objects/s recovering 2024-04-02T04:52:51.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:50 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:51.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:50 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-02T04:52:51.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:50 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:52:51.334 INFO:teuthology.orchestra.run.smithi053.stdout: 5 247975 9.90 MB/sec execute 481 sec latency 53.413 ms 2024-04-02T04:52:52.158 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:51 smithi053 ceph-mon[67180]: Upgrade: Updating osd.5 2024-04-02T04:52:52.158 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:51 smithi053 ceph-mon[67180]: Deploying daemon osd.5 on smithi122 2024-04-02T04:52:52.158 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:51 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:52.334 INFO:teuthology.orchestra.run.smithi053.stdout: 5 248329 9.91 MB/sec execute 482 sec latency 47.131 ms 2024-04-02T04:52:52.357 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:51 smithi122 ceph-mon[53211]: Upgrade: Updating osd.5 2024-04-02T04:52:52.357 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:51 smithi122 ceph-mon[53211]: Deploying daemon osd.5 on smithi122 2024-04-02T04:52:52.357 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:51 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:53.189 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:52 smithi053 ceph-mon[67180]: pgmap v209: 97 pgs: 97 active+clean; 334 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.5 MiB/s wr, 426 op/s; 0 B/s, 2 objects/s recovering 2024-04-02T04:52:53.245 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:52 smithi122 ceph-mon[53211]: pgmap v209: 97 pgs: 97 active+clean; 334 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.5 MiB/s wr, 426 op/s; 0 B/s, 2 objects/s recovering 2024-04-02T04:52:53.334 INFO:teuthology.orchestra.run.smithi053.stdout: 5 248491 9.90 MB/sec execute 483 sec latency 65.008 ms 2024-04-02T04:52:54.337 INFO:teuthology.orchestra.run.smithi053.stdout: 5 248765 9.89 MB/sec execute 484 sec latency 45.720 ms 2024-04-02T04:52:54.645 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:54 smithi122 systemd[1]: Stopping Ceph osd.5 for 48e16742-f0aa-11ee-b647-cb9ed24678a4... 2024-04-02T04:52:55.102 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:54 smithi053 ceph-mon[67180]: pgmap v210: 97 pgs: 97 active+clean; 335 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 4.5 MiB/s wr, 426 op/s; 0 B/s, 2 objects/s recovering 2024-04-02T04:52:55.102 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:54 smithi053 ceph-mon[67180]: osd.5 marked itself down and dead 2024-04-02T04:52:55.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:54 smithi122 ceph-mon[53211]: pgmap v210: 97 pgs: 97 active+clean; 335 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 4.5 MiB/s wr, 426 op/s; 0 B/s, 2 objects/s recovering 2024-04-02T04:52:55.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:54 smithi122 ceph-mon[53211]: osd.5 marked itself down and dead 2024-04-02T04:52:55.145 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:54 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[34379]: 2024-04-02T04:52:54.792+0000 7f0e511e9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T04:52:55.145 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:54 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[34379]: 2024-04-02T04:52:54.792+0000 7f0e511e9700 -1 osd.5 121 *** Got signal Terminated *** 2024-04-02T04:52:55.145 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:54 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[34379]: 2024-04-02T04:52:54.792+0000 7f0e511e9700 -1 osd.5 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T04:52:55.337 INFO:teuthology.orchestra.run.smithi053.stdout: 5 249063 9.88 MB/sec execute 485 sec latency 322.790 ms 2024-04-02T04:52:55.895 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:55 smithi122 podman[59904]: 2024-04-02 04:52:55.629535763 +0000 UTC m=+0.967452152 container died e687a2f6da4dbcd8cb892b166693fb34a3fead270516a4e648d755ee2ebeb8f6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5, distribution-scope=public, version=8, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, ceph=True, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, architecture=x86_64, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-02T04:52:56.252 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:55 smithi053 ceph-mon[67180]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-02T04:52:56.252 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:55 smithi053 ceph-mon[67180]: failed to encode map e122 with expected crc 2024-04-02T04:52:56.252 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:55 smithi053 ceph-mon[67180]: failed to encode map e122 with expected crc 2024-04-02T04:52:56.252 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:55 smithi053 ceph-mon[67180]: osdmap e122: 8 total, 7 up, 8 in 2024-04-02T04:52:56.252 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:55 smithi053 ceph-mon[67180]: failed to encode map e122 with expected crc 2024-04-02T04:52:56.252 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:55 smithi053 ceph-mon[67180]: failed to encode map e122 with expected crc 2024-04-02T04:52:56.337 INFO:teuthology.orchestra.run.smithi053.stdout: 5 249132 9.86 MB/sec execute 486 sec latency 1053.335 ms 2024-04-02T04:52:56.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:55 smithi122 ceph-mon[53211]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-02T04:52:56.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:55 smithi122 ceph-mon[53211]: failed to encode map e122 with expected crc 2024-04-02T04:52:56.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:55 smithi122 ceph-mon[53211]: failed to encode map e122 with expected crc 2024-04-02T04:52:56.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:55 smithi122 ceph-mon[53211]: osdmap e122: 8 total, 7 up, 8 in 2024-04-02T04:52:56.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:55 smithi122 ceph-mon[53211]: failed to encode map e122 with expected crc 2024-04-02T04:52:56.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:55 smithi122 ceph-mon[53211]: failed to encode map e122 with expected crc 2024-04-02T04:52:56.395 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:55 smithi122 podman[59904]: 2024-04-02 04:52:55.976911466 +0000 UTC m=+1.314827843 container cleanup e687a2f6da4dbcd8cb892b166693fb34a3fead270516a4e648d755ee2ebeb8f6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5, release=754, ceph=True, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., architecture=x86_64, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8) 2024-04-02T04:52:56.395 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:55 smithi122 bash[59904]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5 2024-04-02T04:52:56.395 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:56 smithi122 podman[59924]: 2024-04-02 04:52:56.203704694 +0000 UTC m=+0.569371575 container remove e687a2f6da4dbcd8cb892b166693fb34a3fead270516a4e648d755ee2ebeb8f6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, architecture=x86_64, RELEASE=HEAD, vcs-type=git, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, ceph=True, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, distribution-scope=public) 2024-04-02T04:52:56.970 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:56 smithi122 podman[60012]: 2024-04-02 04:52:56.551021991 +0000 UTC m=+0.021198323 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:52:56.970 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:56 smithi122 podman[60012]: 2024-04-02 04:52:56.663094006 +0000 UTC m=+0.133270323 container create 954441b461d86f386488ca9db9f48c32f589185030d5de3ebdd90972b253d320 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-6cec793, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-02T04:52:57.022 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:56 smithi053 ceph-mon[67180]: pgmap v212: 97 pgs: 4 peering, 11 stale+active+clean, 82 active+clean; 333 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.5 MiB/s wr, 446 op/s 2024-04-02T04:52:57.022 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:56 smithi053 ceph-mon[67180]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-02T04:52:57.022 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:56 smithi053 ceph-mon[67180]: failed to encode map e123 with expected crc 2024-04-02T04:52:57.022 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:56 smithi053 ceph-mon[67180]: osdmap e123: 8 total, 7 up, 8 in 2024-04-02T04:52:57.022 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:56 smithi053 ceph-mon[67180]: failed to encode map e123 with expected crc 2024-04-02T04:52:57.022 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:56 smithi053 ceph-mon[67180]: failed to encode map e123 with expected crc 2024-04-02T04:52:57.022 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:56 smithi053 ceph-mon[67180]: failed to encode map e123 with expected crc 2024-04-02T04:52:57.022 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:56 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:57.023 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:56 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:52:57.337 INFO:teuthology.orchestra.run.smithi053.stdout: 5 249403 9.86 MB/sec execute 487 sec latency 72.567 ms 2024-04-02T04:52:57.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:56 smithi122 ceph-mon[53211]: pgmap v212: 97 pgs: 4 peering, 11 stale+active+clean, 82 active+clean; 333 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.5 MiB/s wr, 446 op/s 2024-04-02T04:52:57.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:56 smithi122 ceph-mon[53211]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-02T04:52:57.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:56 smithi122 ceph-mon[53211]: failed to encode map e123 with expected crc 2024-04-02T04:52:57.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:56 smithi122 ceph-mon[53211]: osdmap e123: 8 total, 7 up, 8 in 2024-04-02T04:52:57.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:56 smithi122 ceph-mon[53211]: failed to encode map e123 with expected crc 2024-04-02T04:52:57.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:56 smithi122 ceph-mon[53211]: failed to encode map e123 with expected crc 2024-04-02T04:52:57.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:56 smithi122 ceph-mon[53211]: failed to encode map e123 with expected crc 2024-04-02T04:52:57.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:56 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:52:57.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:56 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:52:57.396 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:57 smithi122 podman[60012]: 2024-04-02 04:52:57.107873638 +0000 UTC m=+0.578049955 container init 954441b461d86f386488ca9db9f48c32f589185030d5de3ebdd90972b253d320 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-6cec793, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-02T04:52:57.396 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:57 smithi122 podman[60012]: 2024-04-02 04:52:57.11186253 +0000 UTC m=+0.582038847 container start 954441b461d86f386488ca9db9f48c32f589185030d5de3ebdd90972b253d320 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-6cec793) 2024-04-02T04:52:57.396 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:57 smithi122 podman[60012]: 2024-04-02 04:52:57.207745197 +0000 UTC m=+0.677921515 container attach 954441b461d86f386488ca9db9f48c32f589185030d5de3ebdd90972b253d320 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-6cec793, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-02T04:52:57.714 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:57 smithi122 podman[60012]: 2024-04-02 04:52:57.411773468 +0000 UTC m=+0.881949782 container died 954441b461d86f386488ca9db9f48c32f589185030d5de3ebdd90972b253d320 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-6cec793, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-02T04:52:58.291 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:58 smithi053 ceph-mon[67180]: Health check failed: Degraded data redundancy: 63/1560 objects degraded (4.038%), 10 pgs degraded (PG_DEGRADED) 2024-04-02T04:52:58.337 INFO:teuthology.orchestra.run.smithi053.stdout: 5 249618 9.85 MB/sec execute 488 sec latency 46.034 ms 2024-04-02T04:52:58.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:58 smithi122 ceph-mon[53211]: Health check failed: Degraded data redundancy: 63/1560 objects degraded (4.038%), 10 pgs degraded (PG_DEGRADED) 2024-04-02T04:52:58.395 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:57 smithi122 podman[60040]: 2024-04-02 04:52:57.995730475 +0000 UTC m=+0.578789891 container remove 954441b461d86f386488ca9db9f48c32f589185030d5de3ebdd90972b253d320 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-6cec793, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=) 2024-04-02T04:52:58.395 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:58 smithi122 systemd[1]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.5.service: Deactivated successfully. 2024-04-02T04:52:58.395 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:58 smithi122 systemd[1]: Stopped Ceph osd.5 for 48e16742-f0aa-11ee-b647-cb9ed24678a4. 2024-04-02T04:52:58.395 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:58 smithi122 systemd[1]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.5.service: Consumed 4min 48.963s CPU time. 2024-04-02T04:52:58.395 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:58 smithi122 systemd[1]: Starting Ceph osd.5 for 48e16742-f0aa-11ee-b647-cb9ed24678a4... 2024-04-02T04:52:58.703 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:58 smithi122 podman[60144]: 2024-04-02 04:52:58.347638011 +0000 UTC m=+0.017906794 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:52:58.703 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:58 smithi122 podman[60144]: 2024-04-02 04:52:58.464169981 +0000 UTC m=+0.134438783 container create 142dc69c47bb2b67ae37f7163e78b4c684d866681c7a925e59bea0b745eaf911 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5-activate, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-6cec793, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-02T04:52:58.704 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:58 smithi122 podman[60144]: 2024-04-02 04:52:58.700307947 +0000 UTC m=+0.370576719 container init 142dc69c47bb2b67ae37f7163e78b4c684d866681c7a925e59bea0b745eaf911 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5-activate, RELEASE=squid-6cec793, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240326) 2024-04-02T04:52:59.051 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:58 smithi122 podman[60144]: 2024-04-02 04:52:58.703636316 +0000 UTC m=+0.373905077 container start 142dc69c47bb2b67ae37f7163e78b4c684d866681c7a925e59bea0b745eaf911 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-6cec793, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-02T04:52:59.052 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:58 smithi122 podman[60144]: 2024-04-02 04:52:58.775268495 +0000 UTC m=+0.445537272 container attach 142dc69c47bb2b67ae37f7163e78b4c684d866681c7a925e59bea0b745eaf911 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5-activate, io.buildah.version=1.33.5, RELEASE=squid-6cec793, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T04:52:59.066 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:52:59 smithi053 ceph-mon[67180]: pgmap v214: 97 pgs: 5 active+undersized, 10 active+undersized+degraded, 4 peering, 6 stale+active+clean, 72 active+clean; 330 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.1 MiB/s wr, 405 op/s; 63/1560 objects degraded (4.038%) 2024-04-02T04:52:59.337 INFO:teuthology.orchestra.run.smithi053.stdout: 5 249942 9.85 MB/sec execute 489 sec latency 74.909 ms 2024-04-02T04:52:59.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:52:59 smithi122 ceph-mon[53211]: pgmap v214: 97 pgs: 5 active+undersized, 10 active+undersized+degraded, 4 peering, 6 stale+active+clean, 72 active+clean; 330 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.1 MiB/s wr, 405 op/s; 63/1560 objects degraded (4.038%) 2024-04-02T04:52:59.395 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:59 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5-activate[60159]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-02T04:52:59.395 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:59 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5-activate[60159]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-02T04:52:59.395 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:59 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5-activate[60159]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-02T04:52:59.395 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:59 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5-activate[60159]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-02T04:52:59.395 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:59 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5-activate[60159]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-04-02T04:52:59.395 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:59 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5-activate[60159]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-02T04:52:59.395 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:59 smithi122 bash[60144]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-02T04:52:59.395 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:59 smithi122 bash[60144]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-02T04:52:59.395 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:59 smithi122 bash[60144]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-02T04:52:59.395 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:59 smithi122 bash[60144]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-02T04:52:59.395 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:59 smithi122 bash[60144]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-04-02T04:52:59.395 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:59 smithi122 bash[60144]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-02T04:52:59.396 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:59 smithi122 bash[60144]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-02T04:52:59.396 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:59 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5-activate[60159]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-02T04:52:59.396 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:59 smithi122 podman[60144]: 2024-04-02 04:52:59.183350671 +0000 UTC m=+0.853619447 container died 142dc69c47bb2b67ae37f7163e78b4c684d866681c7a925e59bea0b745eaf911 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5-activate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-6cec793, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-02T04:53:00.145 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:52:59 smithi122 podman[60291]: 2024-04-02 04:52:59.788352859 +0000 UTC m=+0.599723377 container remove 142dc69c47bb2b67ae37f7163e78b4c684d866681c7a925e59bea0b745eaf911 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5-activate, RELEASE=squid-6cec793, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True) 2024-04-02T04:53:00.145 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:53:00 smithi122 podman[60353]: 2024-04-02 04:52:59.904307518 +0000 UTC m=+0.018245572 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:53:00.145 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:53:00 smithi122 podman[60353]: 2024-04-02 04:53:00.02268979 +0000 UTC m=+0.136627836 container create be26f425f51b969fc42f4f1a6fa98457e1c2d6884f6207536bb0ead8d1231b7e (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-6cec793, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240326) 2024-04-02T04:53:00.338 INFO:teuthology.orchestra.run.smithi053.stdout: 5 250262 9.84 MB/sec execute 490 sec latency 90.877 ms 2024-04-02T04:53:00.645 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:53:00 smithi122 podman[60353]: 2024-04-02 04:53:00.22546053 +0000 UTC m=+0.339398580 container init be26f425f51b969fc42f4f1a6fa98457e1c2d6884f6207536bb0ead8d1231b7e (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, RELEASE=squid-6cec793, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T04:53:00.645 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:53:00 smithi122 podman[60353]: 2024-04-02 04:53:00.228589909 +0000 UTC m=+0.342527955 container start be26f425f51b969fc42f4f1a6fa98457e1c2d6884f6207536bb0ead8d1231b7e (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, RELEASE=squid-6cec793, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T04:53:00.645 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:53:00 smithi122 ceph-osd[60372]: -- 172.21.15.122:0/1069080214 <== mon.1 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5631a5e3b680 con 0x5631a5e05000 2024-04-02T04:53:00.645 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:53:00 smithi122 bash[60353]: be26f425f51b969fc42f4f1a6fa98457e1c2d6884f6207536bb0ead8d1231b7e 2024-04-02T04:53:00.645 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:53:00 smithi122 systemd[1]: Started Ceph osd.5 for 48e16742-f0aa-11ee-b647-cb9ed24678a4. 2024-04-02T04:53:01.145 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:53:00 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:53:00.768+0000 7f45ccb2b7c0 -1 Falling back to public interface 2024-04-02T04:53:01.337 INFO:teuthology.orchestra.run.smithi053.stdout: 5 250550 9.83 MB/sec execute 491 sec latency 46.291 ms 2024-04-02T04:53:01.385 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:01 smithi053 ceph-mon[67180]: pgmap v215: 97 pgs: 7 active+undersized, 20 active+undersized+degraded, 4 peering, 66 active+clean; 326 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.4 MiB/s wr, 338 op/s; 143/1452 objects degraded (9.848%) 2024-04-02T04:53:01.385 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:01.385 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:01.385 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:53:01.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:01 smithi122 ceph-mon[53211]: pgmap v215: 97 pgs: 7 active+undersized, 20 active+undersized+degraded, 4 peering, 66 active+clean; 326 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.4 MiB/s wr, 338 op/s; 143/1452 objects degraded (9.848%) 2024-04-02T04:53:01.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:01.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:01.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:53:02.338 INFO:teuthology.orchestra.run.smithi053.stdout: 5 250757 9.82 MB/sec execute 492 sec latency 65.125 ms 2024-04-02T04:53:02.423 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:02 smithi053 ceph-mon[67180]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-02T04:53:02.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:02 smithi122 ceph-mon[53211]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-02T04:53:03.338 INFO:teuthology.orchestra.run.smithi053.stdout: 5 250986 9.83 MB/sec execute 493 sec latency 50.819 ms 2024-04-02T04:53:03.473 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:03 smithi053 ceph-mon[67180]: pgmap v216: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 330 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.0 KiB/s rd, 2.9 MiB/s wr, 354 op/s; 154/1455 objects degraded (10.584%) 2024-04-02T04:53:03.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:03 smithi122 ceph-mon[53211]: pgmap v216: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 330 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.0 KiB/s rd, 2.9 MiB/s wr, 354 op/s; 154/1455 objects degraded (10.584%) 2024-04-02T04:53:04.338 INFO:teuthology.orchestra.run.smithi053.stdout: 5 251328 9.83 MB/sec execute 494 sec latency 49.275 ms 2024-04-02T04:53:05.296 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:05 smithi053 ceph-mon[67180]: pgmap v217: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 328 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 3.4 MiB/s wr, 372 op/s; 164/1515 objects degraded (10.825%) 2024-04-02T04:53:05.296 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:05 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:05.296 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:05 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:05.296 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:05 smithi053 ceph-mon[67180]: Health check update: Degraded data redundancy: 164/1515 objects degraded (10.825%), 23 pgs degraded (PG_DEGRADED) 2024-04-02T04:53:05.338 INFO:teuthology.orchestra.run.smithi053.stdout: 5 251710 9.83 MB/sec execute 495 sec latency 47.462 ms 2024-04-02T04:53:05.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:05 smithi122 ceph-mon[53211]: pgmap v217: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 328 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 3.4 MiB/s wr, 372 op/s; 164/1515 objects degraded (10.825%) 2024-04-02T04:53:05.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:05 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:05.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:05 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:05.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:05 smithi122 ceph-mon[53211]: Health check update: Degraded data redundancy: 164/1515 objects degraded (10.825%), 23 pgs degraded (PG_DEGRADED) 2024-04-02T04:53:06.338 INFO:teuthology.orchestra.run.smithi053.stdout: 5 251991 9.83 MB/sec execute 496 sec latency 45.394 ms 2024-04-02T04:53:06.739 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:06 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:06.739 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:06 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:06.739 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:06 smithi053 ceph-mon[67180]: pgmap v218: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 331 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.5 MiB/s wr, 394 op/s; 170/1542 objects degraded (11.025%) 2024-04-02T04:53:06.750 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:06 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:06.750 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:06 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:06.750 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:06 smithi122 ceph-mon[53211]: pgmap v218: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 331 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.5 MiB/s wr, 394 op/s; 170/1542 objects degraded (11.025%) 2024-04-02T04:53:07.338 INFO:teuthology.orchestra.run.smithi053.stdout: 5 252163 9.81 MB/sec execute 497 sec latency 63.589 ms 2024-04-02T04:53:08.338 INFO:teuthology.orchestra.run.smithi053.stdout: 5 252585 9.81 MB/sec execute 498 sec latency 41.325 ms 2024-04-02T04:53:08.645 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:53:08 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:53:08.273+0000 7f45ccb2b7c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-04-02T04:53:09.145 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:08 smithi053 ceph-mon[67180]: pgmap v219: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 334 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.0 MiB/s wr, 447 op/s; 170/1575 objects degraded (10.794%) 2024-04-02T04:53:09.338 INFO:teuthology.orchestra.run.smithi053.stdout: 5 252929 9.81 MB/sec execute 499 sec latency 32.146 ms 2024-04-02T04:53:09.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:08 smithi122 ceph-mon[53211]: pgmap v219: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 334 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.0 MiB/s wr, 447 op/s; 170/1575 objects degraded (10.794%) 2024-04-02T04:53:09.894 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:53:09 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:53:09.455+0000 7f45ccb2b7c0 -1 osd.5 121 log_to_monitors true 2024-04-02T04:53:10.162 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:09 smithi053 ceph-mon[67180]: from='osd.5 [v2:172.21.15.122:6808/2186510168,v1:172.21.15.122:6809/2186510168]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-02T04:53:10.162 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:09 smithi053 ceph-mon[67180]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-02T04:53:10.162 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:09 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:10.162 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:09 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:10.162 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:09 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:53:10.162 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:09 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:53:10.163 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:09 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:10.163 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:09 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:10.163 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:09 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:53:10.163 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:09 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:53:10.163 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:09 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:53:10.163 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:09 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:53:10.163 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:09 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-02T04:53:10.338 INFO:teuthology.orchestra.run.smithi053.stdout: 5 253208 9.81 MB/sec execute 500 sec latency 41.401 ms 2024-04-02T04:53:10.394 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:53:10 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:53:10.010+0000 7f45bdf83700 -1 osd.5 121 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T04:53:10.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:09 smithi122 ceph-mon[53211]: from='osd.5 [v2:172.21.15.122:6808/2186510168,v1:172.21.15.122:6809/2186510168]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-02T04:53:10.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:09 smithi122 ceph-mon[53211]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-02T04:53:10.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:09 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:10.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:09 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:10.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:09 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:53:10.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:09 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:53:10.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:09 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:10.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:09 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:10.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:09 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:53:10.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:09 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:53:10.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:09 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:53:10.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:09 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:53:10.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:09 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-02T04:53:11.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:10 smithi053 ceph-mon[67180]: pgmap v220: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 331 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 4.4 MiB/s wr, 430 op/s; 177/1629 objects degraded (10.866%) 2024-04-02T04:53:11.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:10 smithi053 ceph-mon[67180]: pgmap v221: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 331 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.6 MiB/s wr, 477 op/s; 177/1629 objects degraded (10.866%) 2024-04-02T04:53:11.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:10 smithi053 ceph-mon[67180]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-02T04:53:11.183 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:10 smithi053 ceph-mon[67180]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-02T04:53:11.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:10 smithi053 ceph-mon[67180]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-02T04:53:11.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:10 smithi053 ceph-mon[67180]: failed to encode map e124 with expected crc 2024-04-02T04:53:11.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:10 smithi053 ceph-mon[67180]: from='osd.5 [v2:172.21.15.122:6808/2186510168,v1:172.21.15.122:6809/2186510168]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-02T04:53:11.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:10 smithi053 ceph-mon[67180]: osdmap e124: 8 total, 7 up, 8 in 2024-04-02T04:53:11.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:10 smithi053 ceph-mon[67180]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-02T04:53:11.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:10 smithi053 ceph-mon[67180]: failed to encode map e124 with expected crc 2024-04-02T04:53:11.184 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:10 smithi053 ceph-mon[67180]: Health check update: Degraded data redundancy: 177/1629 objects degraded (10.866%), 23 pgs degraded (PG_DEGRADED) 2024-04-02T04:53:11.338 INFO:teuthology.orchestra.run.smithi053.stdout: 5 253533 9.81 MB/sec execute 501 sec latency 303.558 ms 2024-04-02T04:53:11.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:10 smithi122 ceph-mon[53211]: pgmap v220: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 331 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 4.4 MiB/s wr, 430 op/s; 177/1629 objects degraded (10.866%) 2024-04-02T04:53:11.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:10 smithi122 ceph-mon[53211]: pgmap v221: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 331 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.6 MiB/s wr, 477 op/s; 177/1629 objects degraded (10.866%) 2024-04-02T04:53:11.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:10 smithi122 ceph-mon[53211]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-02T04:53:11.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:10 smithi122 ceph-mon[53211]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-02T04:53:11.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:10 smithi122 ceph-mon[53211]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-02T04:53:11.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:10 smithi122 ceph-mon[53211]: failed to encode map e124 with expected crc 2024-04-02T04:53:11.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:11 smithi122 ceph-mon[53211]: from='osd.5 [v2:172.21.15.122:6808/2186510168,v1:172.21.15.122:6809/2186510168]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-02T04:53:11.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:11 smithi122 ceph-mon[53211]: osdmap e124: 8 total, 7 up, 8 in 2024-04-02T04:53:11.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:11 smithi122 ceph-mon[53211]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-02T04:53:11.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:11 smithi122 ceph-mon[53211]: failed to encode map e124 with expected crc 2024-04-02T04:53:11.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:11 smithi122 ceph-mon[53211]: Health check update: Degraded data redundancy: 177/1629 objects degraded (10.866%), 23 pgs degraded (PG_DEGRADED) 2024-04-02T04:53:12.272 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:12 smithi053 ceph-mon[67180]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-02T04:53:12.272 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:12 smithi053 ceph-mon[67180]: osd.5 [v2:172.21.15.122:6808/2186510168,v1:172.21.15.122:6809/2186510168] boot 2024-04-02T04:53:12.272 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:12 smithi053 ceph-mon[67180]: osdmap e125: 8 total, 8 up, 8 in 2024-04-02T04:53:12.272 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:12 smithi053 ceph-mon[67180]: failed to encode map e125 with expected crc 2024-04-02T04:53:12.273 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:12 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:53:12.273 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:12 smithi053 ceph-mon[67180]: failed to encode map e125 with expected crc 2024-04-02T04:53:12.273 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:12 smithi053 ceph-mon[67180]: failed to encode map e125 with expected crc 2024-04-02T04:53:12.273 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:12 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:12.273 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:12 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:53:12.338 INFO:teuthology.orchestra.run.smithi053.stdout: 5 253659 9.79 MB/sec execute 502 sec latency 1041.520 ms 2024-04-02T04:53:12.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:12 smithi122 ceph-mon[53211]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-02T04:53:12.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:12 smithi122 ceph-mon[53211]: osd.5 [v2:172.21.15.122:6808/2186510168,v1:172.21.15.122:6809/2186510168] boot 2024-04-02T04:53:12.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:12 smithi122 ceph-mon[53211]: osdmap e125: 8 total, 8 up, 8 in 2024-04-02T04:53:12.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:12 smithi122 ceph-mon[53211]: failed to encode map e125 with expected crc 2024-04-02T04:53:12.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:12 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T04:53:12.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:12 smithi122 ceph-mon[53211]: failed to encode map e125 with expected crc 2024-04-02T04:53:12.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:12 smithi122 ceph-mon[53211]: failed to encode map e125 with expected crc 2024-04-02T04:53:12.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:12 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:12.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:12 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:53:13.040 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:13 smithi053 ceph-mon[67180]: pgmap v224: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 331 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.0 MiB/s wr, 320 op/s; 177/1629 objects degraded (10.866%) 2024-04-02T04:53:13.041 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:13 smithi053 ceph-mon[67180]: failed to encode map e126 with expected crc 2024-04-02T04:53:13.041 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:13 smithi053 ceph-mon[67180]: osdmap e126: 8 total, 8 up, 8 in 2024-04-02T04:53:13.041 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:13 smithi053 ceph-mon[67180]: failed to encode map e126 with expected crc 2024-04-02T04:53:13.041 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:13 smithi053 ceph-mon[67180]: failed to encode map e126 with expected crc 2024-04-02T04:53:13.041 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:13 smithi053 ceph-mon[67180]: failed to encode map e126 with expected crc 2024-04-02T04:53:13.041 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:13 smithi053 ceph-mon[67180]: failed to encode map e126 with expected crc 2024-04-02T04:53:13.041 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:13 smithi053 ceph-mon[67180]: osdmap e127: 8 total, 8 up, 8 in 2024-04-02T04:53:13.338 INFO:teuthology.orchestra.run.smithi053.stdout: 5 254005 9.78 MB/sec execute 503 sec latency 60.398 ms 2024-04-02T04:53:13.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:13 smithi122 ceph-mon[53211]: pgmap v224: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 331 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.0 MiB/s wr, 320 op/s; 177/1629 objects degraded (10.866%) 2024-04-02T04:53:13.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:13 smithi122 ceph-mon[53211]: failed to encode map e126 with expected crc 2024-04-02T04:53:13.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:13 smithi122 ceph-mon[53211]: osdmap e126: 8 total, 8 up, 8 in 2024-04-02T04:53:13.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:13 smithi122 ceph-mon[53211]: failed to encode map e126 with expected crc 2024-04-02T04:53:13.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:13 smithi122 ceph-mon[53211]: failed to encode map e126 with expected crc 2024-04-02T04:53:13.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:13 smithi122 ceph-mon[53211]: failed to encode map e126 with expected crc 2024-04-02T04:53:13.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:13 smithi122 ceph-mon[53211]: failed to encode map e126 with expected crc 2024-04-02T04:53:13.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:13 smithi122 ceph-mon[53211]: osdmap e127: 8 total, 8 up, 8 in 2024-04-02T04:53:14.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:14 smithi053 ceph-mon[67180]: failed to encode map e127 with expected crc 2024-04-02T04:53:14.338 INFO:teuthology.orchestra.run.smithi053.stdout: 5 254278 9.78 MB/sec execute 504 sec latency 218.023 ms 2024-04-02T04:53:14.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:14 smithi122 ceph-mon[53211]: failed to encode map e127 with expected crc 2024-04-02T04:53:15.079 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:15 smithi053 ceph-mon[67180]: pgmap v227: 97 pgs: 1 active+recovering+degraded, 1 active+recovery_wait+degraded, 1 remapped+peering, 5 active+undersized, 1 active+undersized+remapped, 14 active+undersized+degraded, 74 active+clean; 331 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 2.8 MiB/s wr, 355 op/s; 131/1593 objects degraded (8.223%); 224 KiB/s, 11 objects/s recovering 2024-04-02T04:53:15.079 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:15 smithi053 ceph-mon[67180]: osdmap e128: 8 total, 8 up, 8 in 2024-04-02T04:53:15.079 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:15 smithi053 ceph-mon[67180]: failed to encode map e128 with expected crc 2024-04-02T04:53:15.079 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:15 smithi053 ceph-mon[67180]: failed to encode map e128 with expected crc 2024-04-02T04:53:15.079 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:15 smithi053 ceph-mon[67180]: failed to encode map e128 with expected crc 2024-04-02T04:53:15.338 INFO:teuthology.orchestra.run.smithi053.stdout: 5 254433 9.77 MB/sec execute 505 sec latency 957.409 ms 2024-04-02T04:53:15.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:15 smithi122 ceph-mon[53211]: pgmap v227: 97 pgs: 1 active+recovering+degraded, 1 active+recovery_wait+degraded, 1 remapped+peering, 5 active+undersized, 1 active+undersized+remapped, 14 active+undersized+degraded, 74 active+clean; 331 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 2.8 MiB/s wr, 355 op/s; 131/1593 objects degraded (8.223%); 224 KiB/s, 11 objects/s recovering 2024-04-02T04:53:15.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:15 smithi122 ceph-mon[53211]: osdmap e128: 8 total, 8 up, 8 in 2024-04-02T04:53:15.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:15 smithi122 ceph-mon[53211]: failed to encode map e128 with expected crc 2024-04-02T04:53:15.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:15 smithi122 ceph-mon[53211]: failed to encode map e128 with expected crc 2024-04-02T04:53:15.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:15 smithi122 ceph-mon[53211]: failed to encode map e128 with expected crc 2024-04-02T04:53:15.646 INFO:teuthology.orchestra.run.smithi053.stdout:true 2024-04-02T04:53:16.102 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:16 smithi053 ceph-mon[67180]: failed to encode map e129 with expected crc 2024-04-02T04:53:16.102 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:16 smithi053 ceph-mon[67180]: osdmap e129: 8 total, 8 up, 8 in 2024-04-02T04:53:16.102 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:16 smithi053 ceph-mon[67180]: failed to encode map e129 with expected crc 2024-04-02T04:53:16.102 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:16 smithi053 ceph-mon[67180]: failed to encode map e129 with expected crc 2024-04-02T04:53:16.102 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:16 smithi053 ceph-mon[67180]: Health check update: Degraded data redundancy: 131/1593 objects degraded (8.223%), 16 pgs degraded (PG_DEGRADED) 2024-04-02T04:53:16.339 INFO:teuthology.orchestra.run.smithi053.stdout: 5 254681 9.76 MB/sec execute 506 sec latency 45.229 ms 2024-04-02T04:53:16.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:16 smithi122 ceph-mon[53211]: failed to encode map e129 with expected crc 2024-04-02T04:53:16.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:16 smithi122 ceph-mon[53211]: osdmap e129: 8 total, 8 up, 8 in 2024-04-02T04:53:16.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:16 smithi122 ceph-mon[53211]: failed to encode map e129 with expected crc 2024-04-02T04:53:16.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:16 smithi122 ceph-mon[53211]: failed to encode map e129 with expected crc 2024-04-02T04:53:16.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:16 smithi122 ceph-mon[53211]: Health check update: Degraded data redundancy: 131/1593 objects degraded (8.223%), 16 pgs degraded (PG_DEGRADED) 2024-04-02T04:53:16.502 INFO:teuthology.orchestra.run.smithi053.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-02T04:53:16.503 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager.smithi053 smithi053 *:9093,9094 running (7m) 70s ago 16m 23.6M - 0.23.0 ba2b418f427c bf50c1cd2b3d 2024-04-02T04:53:16.503 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi053 smithi053 running (4m) 70s ago 16m 6660k - 19.0.0-1979-g6cec793d 78f830190839 4ce8915e0157 2024-04-02T04:53:16.503 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi122 smithi122 running (4m) 12s ago 16m 6660k - 19.0.0-1979-g6cec793d 78f830190839 e2b5369e6fdf 2024-04-02T04:53:16.503 INFO:teuthology.orchestra.run.smithi053.stdout:grafana.smithi053 smithi053 *:3000 error 70s ago 16m - - 2024-04-02T04:53:16.503 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi053.nfrfot smithi053 running (11m) 70s ago 10m 37.4M - 17.2.0 e1d6a67b021e c94e9e643514 2024-04-02T04:53:16.503 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi122.mwoyoh smithi122 running (10m) 12s ago 10m 16.2M - 17.2.0 e1d6a67b021e 73376d9ee4b8 2024-04-02T04:53:16.503 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi053.legdmv smithi053 *:8443,9283,8765 running (6m) 70s ago 18m 506M - 19.0.0-1979-g6cec793d 78f830190839 c12d516a31bc 2024-04-02T04:53:16.503 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi122.cpwlyk smithi122 *:8443,9283,8765 running (5m) 12s ago 16m 451M - 19.0.0-1979-g6cec793d 78f830190839 9a545c5e7d81 2024-04-02T04:53:16.503 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi053 smithi053 running (5m) 70s ago 18m 50.4M 2048M 19.0.0-1979-g6cec793d 78f830190839 f53a44f19c00 2024-04-02T04:53:16.503 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi122 smithi122 running (5m) 12s ago 16m 38.8M 2048M 19.0.0-1979-g6cec793d 78f830190839 44b200fed3c5 2024-04-02T04:53:16.503 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.0.0.smithi053.mhkdcb smithi053 *:2049 running (10m) 70s ago 10m 296M - 4.0 e1d6a67b021e 07e5da7bd2c5 2024-04-02T04:53:16.503 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.1.0.smithi122.mauvsb smithi122 *:2049 running (10m) 12s ago 10m 84.2M - 4.0 e1d6a67b021e 4f866685005e 2024-04-02T04:53:16.503 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi053 smithi053 *:9100 running (7m) 70s ago 16m 18.4M - 1.3.1 1dbe0e931976 773b71971e67 2024-04-02T04:53:16.503 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi122 smithi122 *:9100 running (7m) 12s ago 15m 18.6M - 1.3.1 1dbe0e931976 475d439c7980 2024-04-02T04:53:16.503 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 smithi053 running (3m) 70s ago 15m 334M 1937M 19.0.0-1979-g6cec793d 78f830190839 490c346c7639 2024-04-02T04:53:16.504 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 smithi053 running (2m) 70s ago 14m 283M 1937M 19.0.0-1979-g6cec793d 78f830190839 e70e40949c2f 2024-04-02T04:53:16.504 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 smithi053 running (2m) 70s ago 14m 200M 1937M 19.0.0-1979-g6cec793d 78f830190839 4f053827e0dd 2024-04-02T04:53:16.504 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 smithi053 running (81s) 70s ago 13m 12.7M 1937M 19.0.0-1979-g6cec793d 78f830190839 5cd6de7a6535 2024-04-02T04:53:16.504 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 smithi122 running (52s) 12s ago 13m 275M 2704M 19.0.0-1979-g6cec793d 78f830190839 40d6ab1aac4b 2024-04-02T04:53:16.504 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 smithi122 running (16s) 12s ago 13m 13.0M 2704M 19.0.0-1979-g6cec793d 78f830190839 be26f425f51b 2024-04-02T04:53:16.504 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 smithi122 running (12m) 12s ago 12m 516M 2704M 17.2.0 e1d6a67b021e 9e371df9a839 2024-04-02T04:53:16.504 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 smithi122 running (12m) 12s ago 12m 793M 2704M 17.2.0 e1d6a67b021e 9b3cc398254d 2024-04-02T04:53:16.504 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus.smithi053 smithi053 *:9095 running (5m) 70s ago 15m 69.6M - 2.33.4 514e6a882f6e 1dd55399da10 2024-04-02T04:53:17.023 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:53:17.024 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": { 2024-04-02T04:53:17.024 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:53:17.024 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:53:17.024 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": { 2024-04-02T04:53:17.024 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:53:17.024 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:53:17.024 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": { 2024-04-02T04:53:17.024 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-04-02T04:53:17.024 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 6 2024-04-02T04:53:17.024 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:53:17.024 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": { 2024-04-02T04:53:17.024 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:53:17.024 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:53:17.024 INFO:teuthology.orchestra.run.smithi053.stdout: "overall": { 2024-04-02T04:53:17.024 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-04-02T04:53:17.025 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 10 2024-04-02T04:53:17.025 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-02T04:53:17.025 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:53:17.180 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:17 smithi053 ceph-mon[67180]: pgmap v230: 97 pgs: 1 active+recovering+degraded, 1 active+recovery_wait+degraded, 1 remapped+peering, 1 active+undersized, 1 active+undersized+remapped, 2 active+undersized+degraded, 90 active+clean; 330 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.4 MiB/s wr, 458 op/s; 26/1539 objects degraded (1.689%); 554 KiB/s, 46 objects/s recovering 2024-04-02T04:53:17.180 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:17 smithi053 ceph-mon[67180]: from='client.34356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:53:17.180 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:17 smithi053 ceph-mon[67180]: from='client.34360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:53:17.180 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:17 smithi053 ceph-mon[67180]: from='client.34364 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:53:17.180 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:17 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/526489836' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:53:17.338 INFO:teuthology.orchestra.run.smithi053.stdout: 5 254975 9.76 MB/sec execute 507 sec latency 79.320 ms 2024-04-02T04:53:17.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:17 smithi122 ceph-mon[53211]: pgmap v230: 97 pgs: 1 active+recovering+degraded, 1 active+recovery_wait+degraded, 1 remapped+peering, 1 active+undersized, 1 active+undersized+remapped, 2 active+undersized+degraded, 90 active+clean; 330 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.4 MiB/s wr, 458 op/s; 26/1539 objects degraded (1.689%); 554 KiB/s, 46 objects/s recovering 2024-04-02T04:53:17.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:17 smithi122 ceph-mon[53211]: from='client.34356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:53:17.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:17 smithi122 ceph-mon[53211]: from='client.34360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:53:17.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:17 smithi122 ceph-mon[53211]: from='client.34364 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:53:17.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:17 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/526489836' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:53:17.448 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:53:17.448 INFO:teuthology.orchestra.run.smithi053.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae", 2024-04-02T04:53:17.448 INFO:teuthology.orchestra.run.smithi053.stdout: "in_progress": true, 2024-04-02T04:53:17.448 INFO:teuthology.orchestra.run.smithi053.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-02T04:53:17.448 INFO:teuthology.orchestra.run.smithi053.stdout: "services_complete": [ 2024-04-02T04:53:17.448 INFO:teuthology.orchestra.run.smithi053.stdout: "mon", 2024-04-02T04:53:17.448 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr", 2024-04-02T04:53:17.449 INFO:teuthology.orchestra.run.smithi053.stdout: "crash" 2024-04-02T04:53:17.449 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-04-02T04:53:17.449 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": "12/23 daemons upgraded", 2024-04-02T04:53:17.449 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "Currently upgrading osd daemons", 2024-04-02T04:53:17.449 INFO:teuthology.orchestra.run.smithi053.stdout: "is_paused": false 2024-04-02T04:53:17.449 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:53:17.958 INFO:teuthology.orchestra.run.smithi053.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Degraded data redundancy: 26/1539 objects degraded (1.689%), 4 pgs degraded 2024-04-02T04:53:17.958 INFO:teuthology.orchestra.run.smithi053.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-02T04:53:17.958 INFO:teuthology.orchestra.run.smithi053.stdout: daemon grafana.smithi053 on smithi053 is in error state 2024-04-02T04:53:17.959 INFO:teuthology.orchestra.run.smithi053.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 26/1539 objects degraded (1.689%), 4 pgs degraded 2024-04-02T04:53:17.959 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.b is active+undersized+degraded, acting [2,4] 2024-04-02T04:53:17.959 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.c is active+undersized+degraded, acting [2,0] 2024-04-02T04:53:17.959 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.4 is active+recovery_wait+degraded, acting [1,2,5] 2024-04-02T04:53:17.959 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.10 is active+recovering+degraded, acting [6,0,5] 2024-04-02T04:53:18.203 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:18 smithi053 ceph-mon[67180]: from='client.34372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:53:18.203 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:18 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/1172427819' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:53:18.338 INFO:teuthology.orchestra.run.smithi053.stdout: 5 255179 9.75 MB/sec execute 508 sec latency 44.182 ms 2024-04-02T04:53:18.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:18 smithi122 ceph-mon[53211]: from='client.34372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:53:18.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:18 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/1172427819' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:53:19.254 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:19 smithi053 ceph-mon[67180]: pgmap v231: 97 pgs: 1 active+recovering+degraded, 1 active+recovery_wait+degraded, 1 remapped+peering, 1 active+undersized+remapped, 93 active+clean; 329 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.8 MiB/s wr, 556 op/s; 18/1551 objects degraded (1.161%); 397 KiB/s, 33 objects/s recovering 2024-04-02T04:53:19.338 INFO:teuthology.orchestra.run.smithi053.stdout: 5 255483 9.76 MB/sec execute 509 sec latency 103.780 ms 2024-04-02T04:53:19.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:19 smithi122 ceph-mon[53211]: pgmap v231: 97 pgs: 1 active+recovering+degraded, 1 active+recovery_wait+degraded, 1 remapped+peering, 1 active+undersized+remapped, 93 active+clean; 329 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.8 MiB/s wr, 556 op/s; 18/1551 objects degraded (1.161%); 397 KiB/s, 33 objects/s recovering 2024-04-02T04:53:20.278 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:20 smithi053 ceph-mon[67180]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 18/1551 objects degraded (1.161%), 2 pgs degraded) 2024-04-02T04:53:20.338 INFO:teuthology.orchestra.run.smithi053.stdout: 5 255704 9.76 MB/sec execute 510 sec latency 55.522 ms 2024-04-02T04:53:20.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:20 smithi122 ceph-mon[53211]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 18/1551 objects degraded (1.161%), 2 pgs degraded) 2024-04-02T04:53:21.291 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:21 smithi053 ceph-mon[67180]: pgmap v232: 97 pgs: 97 active+clean; 336 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 4.0 KiB/s rd, 4.5 MiB/s wr, 414 op/s; 980 KiB/s, 27 objects/s recovering 2024-04-02T04:53:21.338 INFO:teuthology.orchestra.run.smithi053.stdout: 5 256055 9.76 MB/sec execute 511 sec latency 40.787 ms 2024-04-02T04:53:21.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:21 smithi122 ceph-mon[53211]: pgmap v232: 97 pgs: 97 active+clean; 336 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 4.0 KiB/s rd, 4.5 MiB/s wr, 414 op/s; 980 KiB/s, 27 objects/s recovering 2024-04-02T04:53:22.338 INFO:teuthology.orchestra.run.smithi053.stdout: 5 256356 9.75 MB/sec execute 512 sec latency 43.759 ms 2024-04-02T04:53:23.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:23 smithi053 ceph-mon[67180]: pgmap v233: 97 pgs: 97 active+clean; 332 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.6 MiB/s wr, 334 op/s; 705 KiB/s, 20 objects/s recovering 2024-04-02T04:53:23.338 INFO:teuthology.orchestra.run.smithi053.stdout: 5 256611 9.74 MB/sec execute 513 sec latency 52.724 ms 2024-04-02T04:53:23.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:23 smithi122 ceph-mon[53211]: pgmap v233: 97 pgs: 97 active+clean; 332 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.6 MiB/s wr, 334 op/s; 705 KiB/s, 20 objects/s recovering 2024-04-02T04:53:24.338 INFO:teuthology.orchestra.run.smithi053.stdout: 5 257032 9.74 MB/sec execute 514 sec latency 41.282 ms 2024-04-02T04:53:25.320 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:25 smithi053 ceph-mon[67180]: pgmap v234: 97 pgs: 97 active+clean; 335 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.0 MiB/s wr, 376 op/s; 591 KiB/s, 17 objects/s recovering 2024-04-02T04:53:25.320 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-02T04:53:25.338 INFO:teuthology.orchestra.run.smithi053.stdout: 5 257454 9.74 MB/sec execute 515 sec latency 35.430 ms 2024-04-02T04:53:25.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:25 smithi122 ceph-mon[53211]: pgmap v234: 97 pgs: 97 active+clean; 335 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.0 MiB/s wr, 376 op/s; 591 KiB/s, 17 objects/s recovering 2024-04-02T04:53:25.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-02T04:53:26.338 INFO:teuthology.orchestra.run.smithi053.stdout: 5 257829 9.74 MB/sec execute 516 sec latency 39.099 ms 2024-04-02T04:53:26.343 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:26 smithi053 ceph-mon[67180]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-02T04:53:26.343 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:26 smithi053 ceph-mon[67180]: Upgrade: osd.6 is safe to restart 2024-04-02T04:53:26.535 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:26 smithi122 ceph-mon[53211]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-02T04:53:26.535 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:26 smithi122 ceph-mon[53211]: Upgrade: osd.6 is safe to restart 2024-04-02T04:53:27.338 INFO:teuthology.orchestra.run.smithi053.stdout: 5 258077 9.73 MB/sec execute 517 sec latency 48.724 ms 2024-04-02T04:53:27.451 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:27 smithi053 ceph-mon[67180]: pgmap v235: 97 pgs: 97 active+clean; 339 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.3 MiB/s wr, 444 op/s; 410 KiB/s, 2 objects/s recovering 2024-04-02T04:53:27.451 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:27 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:27.451 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:27 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:53:27.451 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:27 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:27.451 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:27 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-02T04:53:27.451 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:27 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:53:27.451 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:27 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:27.529 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:27 smithi122 ceph-mon[53211]: pgmap v235: 97 pgs: 97 active+clean; 339 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.3 MiB/s wr, 444 op/s; 410 KiB/s, 2 objects/s recovering 2024-04-02T04:53:27.529 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:27 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:27.529 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:27 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:53:27.529 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:27 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:27.529 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:27 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-02T04:53:27.529 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:27 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:53:27.529 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:27 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:28.222 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:28 smithi053 ceph-mon[67180]: Upgrade: Updating osd.6 2024-04-02T04:53:28.222 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:28 smithi053 ceph-mon[67180]: Deploying daemon osd.6 on smithi122 2024-04-02T04:53:28.338 INFO:teuthology.orchestra.run.smithi053.stdout: 5 258438 9.73 MB/sec execute 518 sec latency 45.864 ms 2024-04-02T04:53:28.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:28 smithi122 ceph-mon[53211]: Upgrade: Updating osd.6 2024-04-02T04:53:28.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:28 smithi122 ceph-mon[53211]: Deploying daemon osd.6 on smithi122 2024-04-02T04:53:29.262 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:29 smithi122 ceph-mon[53211]: pgmap v236: 97 pgs: 97 active+clean; 338 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 4.2 MiB/s wr, 408 op/s; 360 KiB/s, 2 objects/s recovering 2024-04-02T04:53:29.304 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:29 smithi053 ceph-mon[67180]: pgmap v236: 97 pgs: 97 active+clean; 338 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 4.2 MiB/s wr, 408 op/s; 360 KiB/s, 2 objects/s recovering 2024-04-02T04:53:29.338 INFO:teuthology.orchestra.run.smithi053.stdout: 5 258712 9.73 MB/sec execute 519 sec latency 68.649 ms 2024-04-02T04:53:30.339 INFO:teuthology.orchestra.run.smithi053.stdout: 5 259040 9.74 MB/sec execute 520 sec latency 48.772 ms 2024-04-02T04:53:30.645 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:30 smithi122 systemd[1]: Stopping Ceph osd.6 for 48e16742-f0aa-11ee-b647-cb9ed24678a4... 2024-04-02T04:53:31.219 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:30 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[37374]: 2024-04-02T04:53:30.926+0000 7f76a17e3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T04:53:31.219 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:30 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[37374]: 2024-04-02T04:53:30.926+0000 7f76a17e3700 -1 osd.6 129 *** Got signal Terminated *** 2024-04-02T04:53:31.219 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:30 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[37374]: 2024-04-02T04:53:30.926+0000 7f76a17e3700 -1 osd.6 129 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T04:53:31.339 INFO:teuthology.orchestra.run.smithi053.stdout: 5 259273 9.74 MB/sec execute 521 sec latency 116.364 ms 2024-04-02T04:53:31.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:31 smithi053 ceph-mon[67180]: pgmap v237: 97 pgs: 97 active+clean; 341 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 4.7 MiB/s wr, 444 op/s; 360 KiB/s, 2 objects/s recovering 2024-04-02T04:53:31.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:31 smithi053 ceph-mon[67180]: osd.6 marked itself down and dead 2024-04-02T04:53:31.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:31 smithi122 ceph-mon[53211]: pgmap v237: 97 pgs: 97 active+clean; 341 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 4.7 MiB/s wr, 444 op/s; 360 KiB/s, 2 objects/s recovering 2024-04-02T04:53:31.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:31 smithi122 ceph-mon[53211]: osd.6 marked itself down and dead 2024-04-02T04:53:32.145 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:31 smithi122 podman[62191]: 2024-04-02 04:53:31.862977479 +0000 UTC m=+1.075617890 container died 9e371df9a8398588a267cb936c172eae19f7ceaf985d0596aa0786b7a7c0c90f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, name=centos-stream, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , architecture=x86_64, build-date=2022-05-03T08:36:31.336870, version=8, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, com.redhat.component=centos-stream-container, RELEASE=HEAD, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream) 2024-04-02T04:53:32.338 INFO:teuthology.orchestra.run.smithi053.stdout: 5 259289 9.72 MB/sec execute 522 sec latency 1058.611 ms 2024-04-02T04:53:32.468 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:32 smithi053 ceph-mon[67180]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-02T04:53:32.468 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:32 smithi053 ceph-mon[67180]: osdmap e130: 8 total, 7 up, 8 in 2024-04-02T04:53:32.468 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:32 smithi053 ceph-mon[67180]: failed to encode map e130 with expected crc 2024-04-02T04:53:32.468 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:32 smithi053 ceph-mon[67180]: failed to encode map e130 with expected crc 2024-04-02T04:53:32.468 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:32 smithi053 ceph-mon[67180]: failed to encode map e130 with expected crc 2024-04-02T04:53:32.468 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:32 smithi053 ceph-mon[67180]: failed to encode map e130 with expected crc 2024-04-02T04:53:32.498 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:32 smithi122 ceph-mon[53211]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-02T04:53:32.498 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:32 smithi122 ceph-mon[53211]: osdmap e130: 8 total, 7 up, 8 in 2024-04-02T04:53:32.498 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:32 smithi122 ceph-mon[53211]: failed to encode map e130 with expected crc 2024-04-02T04:53:32.498 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:32 smithi122 ceph-mon[53211]: failed to encode map e130 with expected crc 2024-04-02T04:53:32.499 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:32 smithi122 ceph-mon[53211]: failed to encode map e130 with expected crc 2024-04-02T04:53:32.499 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:32 smithi122 ceph-mon[53211]: failed to encode map e130 with expected crc 2024-04-02T04:53:32.499 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:32 smithi122 podman[62191]: 2024-04-02 04:53:32.227139089 +0000 UTC m=+1.439779485 container cleanup 9e371df9a8398588a267cb936c172eae19f7ceaf985d0596aa0786b7a7c0c90f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6, io.k8s.display-name=CentOS Stream 8, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, name=centos-stream, RELEASE=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, GIT_CLEAN=True, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, io.openshift.expose-services=, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-02T04:53:32.499 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:32 smithi122 bash[62191]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6 2024-04-02T04:53:32.845 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:32 smithi122 podman[62209]: 2024-04-02 04:53:32.49540545 +0000 UTC m=+0.632757719 container remove 9e371df9a8398588a267cb936c172eae19f7ceaf985d0596aa0786b7a7c0c90f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, architecture=x86_64, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, com.redhat.component=centos-stream-container, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, GIT_CLEAN=True, release=754, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream) 2024-04-02T04:53:33.102 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:32 smithi122 podman[62300]: 2024-04-02 04:53:32.745522624 +0000 UTC m=+0.020383662 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:53:33.102 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:32 smithi122 podman[62300]: 2024-04-02 04:53:32.854746674 +0000 UTC m=+0.129607706 container create 20fa8ed3e9754c43a96f05db91b2a49a45238852152b93eaff42e03568fb6ce0 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-6cec793, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T04:53:33.102 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:33 smithi122 podman[62300]: 2024-04-02 04:53:33.099316129 +0000 UTC m=+0.374177159 container init 20fa8ed3e9754c43a96f05db91b2a49a45238852152b93eaff42e03568fb6ce0 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-6cec793, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-02T04:53:33.246 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:33 smithi053 ceph-mon[67180]: pgmap v239: 97 pgs: 10 stale+active+clean, 87 active+clean; 345 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.8 MiB/s wr, 541 op/s 2024-04-02T04:53:33.246 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:33 smithi053 ceph-mon[67180]: failed to encode map e131 with expected crc 2024-04-02T04:53:33.246 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:33 smithi053 ceph-mon[67180]: osdmap e131: 8 total, 7 up, 8 in 2024-04-02T04:53:33.246 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:33 smithi053 ceph-mon[67180]: failed to encode map e131 with expected crc 2024-04-02T04:53:33.246 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:33 smithi053 ceph-mon[67180]: failed to encode map e131 with expected crc 2024-04-02T04:53:33.246 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:33 smithi053 ceph-mon[67180]: failed to encode map e131 with expected crc 2024-04-02T04:53:33.246 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:33 smithi053 ceph-mon[67180]: failed to encode map e131 with expected crc 2024-04-02T04:53:33.246 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:33 smithi053 ceph-mon[67180]: failed to encode map e131 with expected crc 2024-04-02T04:53:33.246 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:33 smithi053 ceph-mon[67180]: failed to encode map e131 with expected crc 2024-04-02T04:53:33.339 INFO:teuthology.orchestra.run.smithi053.stdout: 5 259526 9.72 MB/sec execute 523 sec latency 55.997 ms 2024-04-02T04:53:33.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:33 smithi122 ceph-mon[53211]: pgmap v239: 97 pgs: 10 stale+active+clean, 87 active+clean; 345 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.8 MiB/s wr, 541 op/s 2024-04-02T04:53:33.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:33 smithi122 ceph-mon[53211]: failed to encode map e131 with expected crc 2024-04-02T04:53:33.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:33 smithi122 ceph-mon[53211]: osdmap e131: 8 total, 7 up, 8 in 2024-04-02T04:53:33.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:33 smithi122 ceph-mon[53211]: failed to encode map e131 with expected crc 2024-04-02T04:53:33.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:33 smithi122 ceph-mon[53211]: failed to encode map e131 with expected crc 2024-04-02T04:53:33.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:33 smithi122 ceph-mon[53211]: failed to encode map e131 with expected crc 2024-04-02T04:53:33.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:33 smithi122 ceph-mon[53211]: failed to encode map e131 with expected crc 2024-04-02T04:53:33.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:33 smithi122 ceph-mon[53211]: failed to encode map e131 with expected crc 2024-04-02T04:53:33.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:33 smithi122 ceph-mon[53211]: failed to encode map e131 with expected crc 2024-04-02T04:53:33.395 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:33 smithi122 podman[62300]: 2024-04-02 04:53:33.102735165 +0000 UTC m=+0.377596204 container start 20fa8ed3e9754c43a96f05db91b2a49a45238852152b93eaff42e03568fb6ce0 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-6cec793, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T04:53:33.396 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:33 smithi122 podman[62300]: 2024-04-02 04:53:33.191084219 +0000 UTC m=+0.465945267 container attach 20fa8ed3e9754c43a96f05db91b2a49a45238852152b93eaff42e03568fb6ce0 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-6cec793, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T04:53:33.895 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:33 smithi122 podman[62300]: 2024-04-02 04:53:33.44813554 +0000 UTC m=+0.722996591 container died 20fa8ed3e9754c43a96f05db91b2a49a45238852152b93eaff42e03568fb6ce0 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240326, RELEASE=squid-6cec793, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T04:53:34.339 INFO:teuthology.orchestra.run.smithi053.stdout: 5 259778 9.70 MB/sec execute 524 sec latency 42.617 ms 2024-04-02T04:53:34.511 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:34 smithi053 ceph-mon[67180]: Health check failed: Degraded data redundancy: 99/1635 objects degraded (6.055%), 12 pgs degraded (PG_DEGRADED) 2024-04-02T04:53:34.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:34 smithi122 ceph-mon[53211]: Health check failed: Degraded data redundancy: 99/1635 objects degraded (6.055%), 12 pgs degraded (PG_DEGRADED) 2024-04-02T04:53:34.645 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:34 smithi122 podman[62325]: 2024-04-02 04:53:34.187397655 +0000 UTC m=+0.733217013 container remove 20fa8ed3e9754c43a96f05db91b2a49a45238852152b93eaff42e03568fb6ce0 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-6cec793, org.label-schema.build-date=20240326) 2024-04-02T04:53:34.645 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:34 smithi122 systemd[1]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.6.service: Deactivated successfully. 2024-04-02T04:53:34.645 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:34 smithi122 systemd[1]: Stopped Ceph osd.6 for 48e16742-f0aa-11ee-b647-cb9ed24678a4. 2024-04-02T04:53:34.645 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:34 smithi122 systemd[1]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.6.service: Consumed 4min 27.281s CPU time. 2024-04-02T04:53:34.646 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:34 smithi122 systemd[1]: Starting Ceph osd.6 for 48e16742-f0aa-11ee-b647-cb9ed24678a4... 2024-04-02T04:53:34.953 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:34 smithi122 podman[62430]: 2024-04-02 04:53:34.562636505 +0000 UTC m=+0.021134965 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:53:34.953 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:34 smithi122 podman[62430]: 2024-04-02 04:53:34.697379659 +0000 UTC m=+0.155878093 container create 90dd5e3ea6396677b9fd0296bc83282388c8563739cba4de81cac4ac184653f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6-activate, RELEASE=squid-6cec793, org.label-schema.build-date=20240326, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T04:53:34.953 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:34 smithi122 podman[62430]: 2024-04-02 04:53:34.950402835 +0000 UTC m=+0.408901271 container init 90dd5e3ea6396677b9fd0296bc83282388c8563739cba4de81cac4ac184653f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6-activate, GIT_BRANCH=HEAD, RELEASE=squid-6cec793, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-02T04:53:35.284 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:34 smithi122 podman[62430]: 2024-04-02 04:53:34.953770751 +0000 UTC m=+0.412269192 container start 90dd5e3ea6396677b9fd0296bc83282388c8563739cba4de81cac4ac184653f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6-activate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-6cec793, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2) 2024-04-02T04:53:35.284 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:35 smithi122 podman[62430]: 2024-04-02 04:53:35.067022818 +0000 UTC m=+0.525521265 container attach 90dd5e3ea6396677b9fd0296bc83282388c8563739cba4de81cac4ac184653f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6-activate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-6cec793, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T04:53:35.339 INFO:teuthology.orchestra.run.smithi053.stdout: 5 260006 9.69 MB/sec execute 525 sec latency 57.436 ms 2024-04-02T04:53:35.535 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:35 smithi053 ceph-mon[67180]: pgmap v241: 97 pgs: 3 active+undersized, 12 active+undersized+degraded, 8 stale+active+clean, 74 active+clean; 336 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.1 MiB/s wr, 410 op/s; 99/1635 objects degraded (6.055%) 2024-04-02T04:53:35.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:35 smithi122 ceph-mon[53211]: pgmap v241: 97 pgs: 3 active+undersized, 12 active+undersized+degraded, 8 stale+active+clean, 74 active+clean; 336 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.1 MiB/s wr, 410 op/s; 99/1635 objects degraded (6.055%) 2024-04-02T04:53:35.645 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:35 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6-activate[62445]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-02T04:53:35.645 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:35 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6-activate[62445]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-02T04:53:35.645 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:35 smithi122 bash[62430]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-02T04:53:35.645 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:35 smithi122 bash[62430]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-02T04:53:35.645 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:35 smithi122 bash[62430]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-02T04:53:35.645 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:35 smithi122 bash[62430]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-02T04:53:35.645 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:35 smithi122 bash[62430]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-04-02T04:53:35.645 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:35 smithi122 bash[62430]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-02T04:53:35.646 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:35 smithi122 bash[62430]: --> ceph-volume raw activate successful for osd ID: 6 2024-04-02T04:53:35.646 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:35 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6-activate[62445]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-02T04:53:35.646 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:35 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6-activate[62445]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-02T04:53:35.646 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:35 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6-activate[62445]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-04-02T04:53:35.646 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:35 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6-activate[62445]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-02T04:53:35.646 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:35 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6-activate[62445]: --> ceph-volume raw activate successful for osd ID: 6 2024-04-02T04:53:35.646 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:35 smithi122 podman[62430]: 2024-04-02 04:53:35.509772234 +0000 UTC m=+0.968270685 container died 90dd5e3ea6396677b9fd0296bc83282388c8563739cba4de81cac4ac184653f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6-activate, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-6cec793, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-02T04:53:36.339 INFO:teuthology.orchestra.run.smithi053.stdout: 5 260251 9.68 MB/sec execute 526 sec latency 44.199 ms 2024-04-02T04:53:36.392 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:36 smithi122 podman[62577]: 2024-04-02 04:53:36.138650899 +0000 UTC m=+0.620320449 container remove 90dd5e3ea6396677b9fd0296bc83282388c8563739cba4de81cac4ac184653f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6-activate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-6cec793, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T04:53:36.392 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:36 smithi122 podman[62637]: 2024-04-02 04:53:36.271254537 +0000 UTC m=+0.020701262 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:53:36.645 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:36 smithi122 podman[62637]: 2024-04-02 04:53:36.389712701 +0000 UTC m=+0.139159418 container create aab18af876da2ac5f684ac887350f6af9b7d61cd6173a57f1613cddb7989a3f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-6cec793, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0) 2024-04-02T04:53:37.145 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:36 smithi122 podman[62637]: 2024-04-02 04:53:36.659137447 +0000 UTC m=+0.408584157 container init aab18af876da2ac5f684ac887350f6af9b7d61cd6173a57f1613cddb7989a3f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-6cec793, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=) 2024-04-02T04:53:37.145 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:36 smithi122 podman[62637]: 2024-04-02 04:53:36.663018182 +0000 UTC m=+0.412464898 container start aab18af876da2ac5f684ac887350f6af9b7d61cd6173a57f1613cddb7989a3f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-6cec793, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-02T04:53:37.145 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:36 smithi122 ceph-osd[62655]: -- 172.21.15.122:0/4024452629 <== mon.1 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a20d177680 con 0x55a20d140c00 2024-04-02T04:53:37.145 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:36 smithi122 bash[62637]: aab18af876da2ac5f684ac887350f6af9b7d61cd6173a57f1613cddb7989a3f6 2024-04-02T04:53:37.145 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:36 smithi122 systemd[1]: Started Ceph osd.6 for 48e16742-f0aa-11ee-b647-cb9ed24678a4. 2024-04-02T04:53:37.339 INFO:teuthology.orchestra.run.smithi053.stdout: 5 260516 9.68 MB/sec execute 527 sec latency 69.831 ms 2024-04-02T04:53:37.357 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:37 smithi053 ceph-mon[67180]: pgmap v242: 97 pgs: 10 active+undersized, 24 active+undersized+degraded, 1 stale+active+clean, 62 active+clean; 343 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 4.9 MiB/s wr, 541 op/s; 197/1653 objects degraded (11.918%) 2024-04-02T04:53:37.357 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:37 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:37.357 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:37 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:37.357 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:37 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:53:37.645 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:37 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:53:37.207+0000 7f28261587c0 -1 Falling back to public interface 2024-04-02T04:53:37.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:37 smithi122 ceph-mon[53211]: pgmap v242: 97 pgs: 10 active+undersized, 24 active+undersized+degraded, 1 stale+active+clean, 62 active+clean; 343 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 4.9 MiB/s wr, 541 op/s; 197/1653 objects degraded (11.918%) 2024-04-02T04:53:37.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:37 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:37.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:37 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:37.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:37 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:53:38.341 INFO:teuthology.orchestra.run.smithi053.stdout: 5 260805 9.68 MB/sec execute 528 sec latency 65.190 ms 2024-04-02T04:53:39.339 INFO:teuthology.orchestra.run.smithi053.stdout: 5 261134 9.67 MB/sec execute 529 sec latency 63.517 ms 2024-04-02T04:53:39.403 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:39 smithi053 ceph-mon[67180]: pgmap v243: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 342 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.6 KiB/s rd, 3.9 MiB/s wr, 431 op/s; 195/1632 objects degraded (11.949%) 2024-04-02T04:53:39.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:39 smithi122 ceph-mon[53211]: pgmap v243: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 342 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.6 KiB/s rd, 3.9 MiB/s wr, 431 op/s; 195/1632 objects degraded (11.949%) 2024-04-02T04:53:40.339 INFO:teuthology.orchestra.run.smithi053.stdout: 5 261467 9.66 MB/sec execute 530 sec latency 47.923 ms 2024-04-02T04:53:40.429 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:40 smithi053 ceph-mon[67180]: Health check update: Degraded data redundancy: 191/1566 objects degraded (12.197%), 25 pgs degraded (PG_DEGRADED) 2024-04-02T04:53:40.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:40 smithi122 ceph-mon[53211]: Health check update: Degraded data redundancy: 191/1566 objects degraded (12.197%), 25 pgs degraded (PG_DEGRADED) 2024-04-02T04:53:41.339 INFO:teuthology.orchestra.run.smithi053.stdout: 5 261818 9.66 MB/sec execute 531 sec latency 45.286 ms 2024-04-02T04:53:41.454 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:41 smithi053 ceph-mon[67180]: pgmap v244: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 333 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 4.0 MiB/s wr, 338 op/s; 191/1566 objects degraded (12.197%) 2024-04-02T04:53:41.454 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:41 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:41.454 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:41 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:41.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:41 smithi122 ceph-mon[53211]: pgmap v244: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 333 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 4.0 MiB/s wr, 338 op/s; 191/1566 objects degraded (12.197%) 2024-04-02T04:53:41.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:41 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:41.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:41 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:42.339 INFO:teuthology.orchestra.run.smithi053.stdout: 5 262094 9.66 MB/sec execute 532 sec latency 44.462 ms 2024-04-02T04:53:42.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:42 smithi053 ceph-mon[67180]: pgmap v245: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 332 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 3.5 MiB/s wr, 318 op/s; 191/1557 objects degraded (12.267%) 2024-04-02T04:53:42.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:42 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:42.732 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:42 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:53:42.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:42 smithi122 ceph-mon[53211]: pgmap v245: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 332 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 3.5 MiB/s wr, 318 op/s; 191/1557 objects degraded (12.267%) 2024-04-02T04:53:42.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:42 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:42.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:42 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:53:43.339 INFO:teuthology.orchestra.run.smithi053.stdout: 5 262415 9.67 MB/sec execute 533 sec latency 53.722 ms 2024-04-02T04:53:43.747 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:43 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:43.747 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:43 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:44.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:43 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:44.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:43 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:44.339 INFO:teuthology.orchestra.run.smithi053.stdout: 5 262870 9.68 MB/sec execute 534 sec latency 49.499 ms 2024-04-02T04:53:44.645 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:44 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:53:44.608+0000 7f28261587c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-04-02T04:53:44.759 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:44 smithi053 ceph-mon[67180]: pgmap v246: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 331 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.1 MiB/s wr, 364 op/s; 194/1578 objects degraded (12.294%) 2024-04-02T04:53:44.982 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:44 smithi122 ceph-mon[53211]: pgmap v246: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 331 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.1 MiB/s wr, 364 op/s; 194/1578 objects degraded (12.294%) 2024-04-02T04:53:45.339 INFO:teuthology.orchestra.run.smithi053.stdout: 5 263272 9.68 MB/sec execute 535 sec latency 39.517 ms 2024-04-02T04:53:45.769 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:45 smithi053 ceph-mon[67180]: Health check update: Degraded data redundancy: 200/1635 objects degraded (12.232%), 25 pgs degraded (PG_DEGRADED) 2024-04-02T04:53:45.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:45 smithi122 ceph-mon[53211]: Health check update: Degraded data redundancy: 200/1635 objects degraded (12.232%), 25 pgs degraded (PG_DEGRADED) 2024-04-02T04:53:45.895 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:45 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:53:45.725+0000 7f28261587c0 -1 osd.6 129 log_to_monitors true 2024-04-02T04:53:46.339 INFO:teuthology.orchestra.run.smithi053.stdout: 5 263548 9.67 MB/sec execute 536 sec latency 45.827 ms 2024-04-02T04:53:46.789 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:46 smithi053 ceph-mon[67180]: pgmap v247: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 339 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 4.4 MiB/s wr, 438 op/s; 200/1635 objects degraded (12.232%) 2024-04-02T04:53:46.789 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:46 smithi053 ceph-mon[67180]: from='osd.6 [v2:172.21.15.122:6816/1555198593,v1:172.21.15.122:6817/1555198593]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-02T04:53:46.789 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:46 smithi053 ceph-mon[67180]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-02T04:53:46.789 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:46 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:46.789 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:46 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:46.790 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:46 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:53:46.790 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:46 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:53:46.790 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:46 smithi053 ceph-mon[67180]: pgmap v248: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 339 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.7 MiB/s wr, 364 op/s; 200/1635 objects degraded (12.232%) 2024-04-02T04:53:46.790 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:46 smithi053 ceph-mon[67180]: pgmap v249: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 339 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 3.9 MiB/s wr, 355 op/s; 200/1635 objects degraded (12.232%) 2024-04-02T04:53:46.790 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:46 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:46.790 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:46 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:46.790 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:46 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:53:46.790 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:46 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:53:46.790 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:46 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:53:46.790 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:46 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:53:46.790 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:46 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-02T04:53:47.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:46 smithi122 ceph-mon[53211]: pgmap v247: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 339 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 4.4 MiB/s wr, 438 op/s; 200/1635 objects degraded (12.232%) 2024-04-02T04:53:47.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:46 smithi122 ceph-mon[53211]: from='osd.6 [v2:172.21.15.122:6816/1555198593,v1:172.21.15.122:6817/1555198593]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-02T04:53:47.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:46 smithi122 ceph-mon[53211]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-02T04:53:47.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:46 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:47.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:46 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:47.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:46 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:53:47.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:46 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:53:47.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:46 smithi122 ceph-mon[53211]: pgmap v248: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 339 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.7 MiB/s wr, 364 op/s; 200/1635 objects degraded (12.232%) 2024-04-02T04:53:47.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:46 smithi122 ceph-mon[53211]: pgmap v249: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 339 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 3.9 MiB/s wr, 355 op/s; 200/1635 objects degraded (12.232%) 2024-04-02T04:53:47.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:46 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:47.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:46 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:47.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:46 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:53:47.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:46 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:53:47.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:46 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:53:47.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:46 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:53:47.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:46 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-02T04:53:47.339 INFO:teuthology.orchestra.run.smithi053.stdout: 5 263677 9.66 MB/sec execute 537 sec latency 180.670 ms 2024-04-02T04:53:47.644 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:53:47 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:53:47.222+0000 7f28135a8700 -1 osd.6 129 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T04:53:47.838 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:47 smithi053 ceph-mon[67180]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-02T04:53:47.838 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:47 smithi053 ceph-mon[67180]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-02T04:53:47.838 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:47 smithi053 ceph-mon[67180]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-02T04:53:47.838 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:47 smithi053 ceph-mon[67180]: osdmap e132: 8 total, 7 up, 8 in 2024-04-02T04:53:47.838 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:47 smithi053 ceph-mon[67180]: from='osd.6 [v2:172.21.15.122:6816/1555198593,v1:172.21.15.122:6817/1555198593]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-02T04:53:47.839 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:47 smithi053 ceph-mon[67180]: failed to encode map e132 with expected crc 2024-04-02T04:53:47.839 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:47 smithi053 ceph-mon[67180]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-02T04:53:47.839 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:47 smithi053 ceph-mon[67180]: from='osd.6 ' entity='osd.6' 2024-04-02T04:53:47.948 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:47 smithi122 ceph-mon[53211]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-02T04:53:47.948 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:47 smithi122 ceph-mon[53211]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-02T04:53:47.948 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:47 smithi122 ceph-mon[53211]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-02T04:53:47.948 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:47 smithi122 ceph-mon[53211]: osdmap e132: 8 total, 7 up, 8 in 2024-04-02T04:53:47.948 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:47 smithi122 ceph-mon[53211]: from='osd.6 [v2:172.21.15.122:6816/1555198593,v1:172.21.15.122:6817/1555198593]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-02T04:53:47.948 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:47 smithi122 ceph-mon[53211]: failed to encode map e132 with expected crc 2024-04-02T04:53:47.949 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:47 smithi122 ceph-mon[53211]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-02T04:53:47.949 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:47 smithi122 ceph-mon[53211]: from='osd.6 ' entity='osd.6' 2024-04-02T04:53:48.339 INFO:teuthology.orchestra.run.smithi053.stdout: 5 263954 9.66 MB/sec execute 538 sec latency 109.993 ms 2024-04-02T04:53:48.504 INFO:teuthology.orchestra.run.smithi053.stdout:true 2024-04-02T04:53:49.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:48 smithi053 ceph-mon[67180]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-02T04:53:49.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:48 smithi053 ceph-mon[67180]: osd.6 [v2:172.21.15.122:6816/1555198593,v1:172.21.15.122:6817/1555198593] boot 2024-04-02T04:53:49.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:48 smithi053 ceph-mon[67180]: osdmap e133: 8 total, 8 up, 8 in 2024-04-02T04:53:49.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:48 smithi053 ceph-mon[67180]: failed to encode map e133 with expected crc 2024-04-02T04:53:49.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:48 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:53:49.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:48 smithi053 ceph-mon[67180]: failed to encode map e133 with expected crc 2024-04-02T04:53:49.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:48 smithi053 ceph-mon[67180]: failed to encode map e133 with expected crc 2024-04-02T04:53:49.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:48 smithi053 ceph-mon[67180]: from='client.34382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:53:49.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:48 smithi053 ceph-mon[67180]: pgmap v252: 97 pgs: 6 peering, 8 active+undersized, 21 active+undersized+degraded, 62 active+clean; 340 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 6.0 MiB/s wr, 653 op/s; 164/1593 objects degraded (10.295%) 2024-04-02T04:53:49.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:48 smithi122 ceph-mon[53211]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-02T04:53:49.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:48 smithi122 ceph-mon[53211]: osd.6 [v2:172.21.15.122:6816/1555198593,v1:172.21.15.122:6817/1555198593] boot 2024-04-02T04:53:49.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:48 smithi122 ceph-mon[53211]: osdmap e133: 8 total, 8 up, 8 in 2024-04-02T04:53:49.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:48 smithi122 ceph-mon[53211]: failed to encode map e133 with expected crc 2024-04-02T04:53:49.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:48 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T04:53:49.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:48 smithi122 ceph-mon[53211]: failed to encode map e133 with expected crc 2024-04-02T04:53:49.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:48 smithi122 ceph-mon[53211]: failed to encode map e133 with expected crc 2024-04-02T04:53:49.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:48 smithi122 ceph-mon[53211]: from='client.34382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:53:49.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:48 smithi122 ceph-mon[53211]: pgmap v252: 97 pgs: 6 peering, 8 active+undersized, 21 active+undersized+degraded, 62 active+clean; 340 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 6.0 MiB/s wr, 653 op/s; 164/1593 objects degraded (10.295%) 2024-04-02T04:53:49.191 INFO:teuthology.orchestra.run.smithi053.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-02T04:53:49.192 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager.smithi053 smithi053 *:9093,9094 running (8m) 103s ago 17m 23.6M - 0.23.0 ba2b418f427c bf50c1cd2b3d 2024-04-02T04:53:49.192 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi053 smithi053 running (4m) 103s ago 17m 6660k - 19.0.0-1979-g6cec793d 78f830190839 4ce8915e0157 2024-04-02T04:53:49.192 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi122 smithi122 running (4m) 7s ago 16m 6660k - 19.0.0-1979-g6cec793d 78f830190839 e2b5369e6fdf 2024-04-02T04:53:49.192 INFO:teuthology.orchestra.run.smithi053.stdout:grafana.smithi053 smithi053 *:3000 error 103s ago 17m - - 2024-04-02T04:53:49.192 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi053.nfrfot smithi053 running (11m) 103s ago 11m 37.4M - 17.2.0 e1d6a67b021e c94e9e643514 2024-04-02T04:53:49.192 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi122.mwoyoh smithi122 running (11m) 7s ago 11m 16.4M - 17.2.0 e1d6a67b021e 73376d9ee4b8 2024-04-02T04:53:49.192 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi053.legdmv smithi053 *:8443,9283,8765 running (7m) 103s ago 18m 506M - 19.0.0-1979-g6cec793d 78f830190839 c12d516a31bc 2024-04-02T04:53:49.192 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi122.cpwlyk smithi122 *:8443,9283,8765 running (6m) 7s ago 16m 451M - 19.0.0-1979-g6cec793d 78f830190839 9a545c5e7d81 2024-04-02T04:53:49.192 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi053 smithi053 running (6m) 103s ago 18m 50.4M 2048M 19.0.0-1979-g6cec793d 78f830190839 f53a44f19c00 2024-04-02T04:53:49.192 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi122 smithi122 running (5m) 7s ago 16m 42.2M 2048M 19.0.0-1979-g6cec793d 78f830190839 44b200fed3c5 2024-04-02T04:53:49.192 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.0.0.smithi053.mhkdcb smithi053 *:2049 running (11m) 103s ago 11m 296M - 4.0 e1d6a67b021e 07e5da7bd2c5 2024-04-02T04:53:49.192 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.1.0.smithi122.mauvsb smithi122 *:2049 running (11m) 7s ago 11m 84.3M - 4.0 e1d6a67b021e 4f866685005e 2024-04-02T04:53:49.192 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi053 smithi053 *:9100 running (7m) 103s ago 17m 18.4M - 1.3.1 1dbe0e931976 773b71971e67 2024-04-02T04:53:49.192 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi122 smithi122 *:9100 running (7m) 7s ago 16m 16.2M - 1.3.1 1dbe0e931976 475d439c7980 2024-04-02T04:53:49.192 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 smithi053 running (4m) 103s ago 15m 334M 1937M 19.0.0-1979-g6cec793d 78f830190839 490c346c7639 2024-04-02T04:53:49.193 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 smithi053 running (3m) 103s ago 15m 283M 1937M 19.0.0-1979-g6cec793d 78f830190839 e70e40949c2f 2024-04-02T04:53:49.193 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 smithi053 running (2m) 103s ago 14m 200M 1937M 19.0.0-1979-g6cec793d 78f830190839 4f053827e0dd 2024-04-02T04:53:49.193 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 smithi053 running (114s) 103s ago 14m 12.7M 1937M 19.0.0-1979-g6cec793d 78f830190839 5cd6de7a6535 2024-04-02T04:53:49.193 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 smithi122 running (85s) 7s ago 14m 313M 2704M 19.0.0-1979-g6cec793d 78f830190839 40d6ab1aac4b 2024-04-02T04:53:49.193 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 smithi122 running (49s) 7s ago 13m 262M 2704M 19.0.0-1979-g6cec793d 78f830190839 be26f425f51b 2024-04-02T04:53:49.193 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 smithi122 running (12s) 7s ago 13m 12.6M 2704M 19.0.0-1979-g6cec793d 78f830190839 aab18af876da 2024-04-02T04:53:49.193 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 smithi122 running (12m) 7s ago 12m 567M 2704M 17.2.0 e1d6a67b021e 9b3cc398254d 2024-04-02T04:53:49.193 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus.smithi053 smithi053 *:9095 running (6m) 103s ago 16m 69.6M - 2.33.4 514e6a882f6e 1dd55399da10 2024-04-02T04:53:49.339 INFO:teuthology.orchestra.run.smithi053.stdout: 5 264000 9.64 MB/sec execute 539 sec latency 1082.622 ms 2024-04-02T04:53:49.691 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:53:49.691 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": { 2024-04-02T04:53:49.692 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:53:49.692 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:53:49.692 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": { 2024-04-02T04:53:49.692 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:53:49.692 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:53:49.692 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": { 2024-04-02T04:53:49.692 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-02T04:53:49.692 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 7 2024-04-02T04:53:49.692 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:53:49.692 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": { 2024-04-02T04:53:49.692 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:53:49.692 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:53:49.692 INFO:teuthology.orchestra.run.smithi053.stdout: "overall": { 2024-04-02T04:53:49.692 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-04-02T04:53:49.692 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 11 2024-04-02T04:53:49.693 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-02T04:53:49.693 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:53:50.149 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:53:50.149 INFO:teuthology.orchestra.run.smithi053.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae", 2024-04-02T04:53:50.149 INFO:teuthology.orchestra.run.smithi053.stdout: "in_progress": true, 2024-04-02T04:53:50.149 INFO:teuthology.orchestra.run.smithi053.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-02T04:53:50.149 INFO:teuthology.orchestra.run.smithi053.stdout: "services_complete": [ 2024-04-02T04:53:50.149 INFO:teuthology.orchestra.run.smithi053.stdout: "mon", 2024-04-02T04:53:50.149 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr", 2024-04-02T04:53:50.150 INFO:teuthology.orchestra.run.smithi053.stdout: "crash" 2024-04-02T04:53:50.150 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-04-02T04:53:50.150 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": "13/23 daemons upgraded", 2024-04-02T04:53:50.150 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "Currently upgrading osd daemons", 2024-04-02T04:53:50.150 INFO:teuthology.orchestra.run.smithi053.stdout: "is_paused": false 2024-04-02T04:53:50.150 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:53:50.339 INFO:teuthology.orchestra.run.smithi053.stdout: 5 264144 9.63 MB/sec execute 540 sec latency 1928.603 ms 2024-04-02T04:53:50.469 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:50 smithi053 ceph-mon[67180]: from='client.34386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:53:50.469 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:50 smithi053 ceph-mon[67180]: from='client.34390 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:53:50.469 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:50 smithi053 ceph-mon[67180]: failed to encode map e134 with expected crc 2024-04-02T04:53:50.469 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:50 smithi053 ceph-mon[67180]: osdmap e134: 8 total, 8 up, 8 in 2024-04-02T04:53:50.470 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:50 smithi053 ceph-mon[67180]: failed to encode map e134 with expected crc 2024-04-02T04:53:50.470 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:50 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/3094588208' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:53:50.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:50 smithi122 ceph-mon[53211]: from='client.34386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:53:50.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:50 smithi122 ceph-mon[53211]: from='client.34390 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:53:50.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:50 smithi122 ceph-mon[53211]: failed to encode map e134 with expected crc 2024-04-02T04:53:50.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:50 smithi122 ceph-mon[53211]: osdmap e134: 8 total, 8 up, 8 in 2024-04-02T04:53:50.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:50 smithi122 ceph-mon[53211]: failed to encode map e134 with expected crc 2024-04-02T04:53:50.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:50 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/3094588208' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:53:50.705 INFO:teuthology.orchestra.run.smithi053.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Degraded data redundancy: 164/1593 objects degraded (10.295%), 21 pgs degraded 2024-04-02T04:53:50.705 INFO:teuthology.orchestra.run.smithi053.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-02T04:53:50.705 INFO:teuthology.orchestra.run.smithi053.stdout: daemon grafana.smithi053 on smithi053 is in error state 2024-04-02T04:53:50.705 INFO:teuthology.orchestra.run.smithi053.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 164/1593 objects degraded (10.295%), 21 pgs degraded 2024-04-02T04:53:50.705 INFO:teuthology.orchestra.run.smithi053.stdout: pg 1.0 is active+undersized+degraded, acting [7,0] 2024-04-02T04:53:50.705 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.2 is active+undersized+degraded, acting [5,1] 2024-04-02T04:53:50.705 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.6 is active+undersized+degraded, acting [1,4] 2024-04-02T04:53:50.706 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.7 is active+undersized+degraded, acting [7,2] 2024-04-02T04:53:50.706 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.11 is active+undersized+degraded, acting [4,1] 2024-04-02T04:53:50.706 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.14 is active+undersized+degraded, acting [3,5] 2024-04-02T04:53:50.706 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.16 is active+undersized+degraded, acting [5,2] 2024-04-02T04:53:50.706 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.17 is active+undersized+degraded, acting [5,2] 2024-04-02T04:53:50.706 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.1a is active+undersized+degraded, acting [4,7] 2024-04-02T04:53:50.706 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.1d is active+undersized+degraded, acting [7,0] 2024-04-02T04:53:50.706 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.0 is active+undersized+degraded, acting [1,2] 2024-04-02T04:53:50.706 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.3 is active+undersized+degraded, acting [4,0] 2024-04-02T04:53:50.706 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.a is active+undersized+degraded, acting [4,1] 2024-04-02T04:53:50.706 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.c is active+undersized+degraded, acting [5,3] 2024-04-02T04:53:50.706 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.d is active+undersized+degraded, acting [7,5] 2024-04-02T04:53:50.706 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.10 is active+undersized+degraded, acting [0,5] 2024-04-02T04:53:50.706 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.11 is active+undersized+degraded, acting [7,4] 2024-04-02T04:53:50.706 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.14 is active+undersized+degraded, acting [4,7] 2024-04-02T04:53:50.707 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.1d is active+undersized+degraded, acting [5,4] 2024-04-02T04:53:50.707 INFO:teuthology.orchestra.run.smithi053.stdout: pg 4.5 is active+undersized+degraded, acting [4,7] 2024-04-02T04:53:50.707 INFO:teuthology.orchestra.run.smithi053.stdout: pg 4.a is active+undersized+degraded, acting [2,7] 2024-04-02T04:53:51.339 INFO:teuthology.orchestra.run.smithi053.stdout: 5 264432 9.62 MB/sec execute 541 sec latency 2064.773 ms 2024-04-02T04:53:51.500 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:51 smithi053 ceph-mon[67180]: from='client.34398 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:53:51.500 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:51 smithi053 ceph-mon[67180]: failed to encode map e135 with expected crc 2024-04-02T04:53:51.500 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:51 smithi053 ceph-mon[67180]: osdmap e135: 8 total, 8 up, 8 in 2024-04-02T04:53:51.500 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:51 smithi053 ceph-mon[67180]: failed to encode map e135 with expected crc 2024-04-02T04:53:51.500 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:51 smithi053 ceph-mon[67180]: pgmap v255: 97 pgs: 3 remapped+peering, 13 peering, 2 active+undersized, 15 active+undersized+degraded, 64 active+clean; 337 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.4 MiB/s wr, 521 op/s; 99/1503 objects degraded (6.587%); 20 KiB/s, 3 objects/s recovering 2024-04-02T04:53:51.500 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:51 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/3221021543' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:53:51.500 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:51 smithi053 ceph-mon[67180]: Health check update: Degraded data redundancy: 164/1593 objects degraded (10.295%), 21 pgs degraded (PG_DEGRADED) 2024-04-02T04:53:51.500 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:51 smithi053 ceph-mon[67180]: osdmap e136: 8 total, 8 up, 8 in 2024-04-02T04:53:51.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:51 smithi122 ceph-mon[53211]: from='client.34398 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:53:51.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:51 smithi122 ceph-mon[53211]: failed to encode map e135 with expected crc 2024-04-02T04:53:51.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:51 smithi122 ceph-mon[53211]: osdmap e135: 8 total, 8 up, 8 in 2024-04-02T04:53:51.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:51 smithi122 ceph-mon[53211]: failed to encode map e135 with expected crc 2024-04-02T04:53:51.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:51 smithi122 ceph-mon[53211]: pgmap v255: 97 pgs: 3 remapped+peering, 13 peering, 2 active+undersized, 15 active+undersized+degraded, 64 active+clean; 337 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.4 MiB/s wr, 521 op/s; 99/1503 objects degraded (6.587%); 20 KiB/s, 3 objects/s recovering 2024-04-02T04:53:51.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:51 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/3221021543' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:53:51.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:51 smithi122 ceph-mon[53211]: Health check update: Degraded data redundancy: 164/1593 objects degraded (10.295%), 21 pgs degraded (PG_DEGRADED) 2024-04-02T04:53:51.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:51 smithi122 ceph-mon[53211]: osdmap e136: 8 total, 8 up, 8 in 2024-04-02T04:53:52.275 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:52 smithi053 ceph-mon[67180]: failed to encode map e136 with expected crc 2024-04-02T04:53:52.275 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:52 smithi053 ceph-mon[67180]: failed to encode map e136 with expected crc 2024-04-02T04:53:52.275 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:52 smithi053 ceph-mon[67180]: failed to encode map e136 with expected crc 2024-04-02T04:53:52.275 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:52 smithi053 ceph-mon[67180]: failed to encode map e136 with expected crc 2024-04-02T04:53:52.275 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:52 smithi053 ceph-mon[67180]: osdmap e137: 8 total, 8 up, 8 in 2024-04-02T04:53:52.339 INFO:teuthology.orchestra.run.smithi053.stdout: 5 264684 9.62 MB/sec execute 542 sec latency 687.944 ms 2024-04-02T04:53:52.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:52 smithi122 ceph-mon[53211]: failed to encode map e136 with expected crc 2024-04-02T04:53:52.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:52 smithi122 ceph-mon[53211]: failed to encode map e136 with expected crc 2024-04-02T04:53:52.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:52 smithi122 ceph-mon[53211]: failed to encode map e136 with expected crc 2024-04-02T04:53:52.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:52 smithi122 ceph-mon[53211]: failed to encode map e136 with expected crc 2024-04-02T04:53:52.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:52 smithi122 ceph-mon[53211]: osdmap e137: 8 total, 8 up, 8 in 2024-04-02T04:53:53.295 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:53 smithi053 ceph-mon[67180]: failed to encode map e137 with expected crc 2024-04-02T04:53:53.295 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:53 smithi053 ceph-mon[67180]: failed to encode map e137 with expected crc 2024-04-02T04:53:53.295 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:53 smithi053 ceph-mon[67180]: failed to encode map e137 with expected crc 2024-04-02T04:53:53.295 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:53 smithi053 ceph-mon[67180]: failed to encode map e137 with expected crc 2024-04-02T04:53:53.295 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:53 smithi053 ceph-mon[67180]: pgmap v258: 97 pgs: 3 remapped+peering, 13 peering, 1 active+undersized, 6 active+undersized+degraded, 74 active+clean; 329 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 1.3 MiB/s wr, 83 op/s; 55/1497 objects degraded (3.674%); 71 KiB/s, 12 objects/s recovering 2024-04-02T04:53:53.295 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:53 smithi053 ceph-mon[67180]: osdmap e138: 8 total, 8 up, 8 in 2024-04-02T04:53:53.339 INFO:teuthology.orchestra.run.smithi053.stdout: 5 264968 9.61 MB/sec execute 543 sec latency 931.850 ms 2024-04-02T04:53:53.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:53 smithi122 ceph-mon[53211]: failed to encode map e137 with expected crc 2024-04-02T04:53:53.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:53 smithi122 ceph-mon[53211]: failed to encode map e137 with expected crc 2024-04-02T04:53:53.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:53 smithi122 ceph-mon[53211]: failed to encode map e137 with expected crc 2024-04-02T04:53:53.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:53 smithi122 ceph-mon[53211]: failed to encode map e137 with expected crc 2024-04-02T04:53:53.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:53 smithi122 ceph-mon[53211]: pgmap v258: 97 pgs: 3 remapped+peering, 13 peering, 1 active+undersized, 6 active+undersized+degraded, 74 active+clean; 329 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 1.3 MiB/s wr, 83 op/s; 55/1497 objects degraded (3.674%); 71 KiB/s, 12 objects/s recovering 2024-04-02T04:53:53.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:53 smithi122 ceph-mon[53211]: osdmap e138: 8 total, 8 up, 8 in 2024-04-02T04:53:54.319 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:54 smithi053 ceph-mon[67180]: failed to encode map e138 with expected crc 2024-04-02T04:53:54.339 INFO:teuthology.orchestra.run.smithi053.stdout: 5 265221 9.60 MB/sec execute 544 sec latency 48.720 ms 2024-04-02T04:53:54.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:54 smithi122 ceph-mon[53211]: failed to encode map e138 with expected crc 2024-04-02T04:53:55.338 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:55 smithi053 ceph-mon[67180]: pgmap v260: 97 pgs: 3 remapped+peering, 1 peering, 93 active+clean; 330 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 2.5 MiB/s wr, 327 op/s; 1.3 MiB/s, 54 keys/s, 37 objects/s recovering 2024-04-02T04:53:55.339 INFO:teuthology.orchestra.run.smithi053.stdout: 5 265466 9.59 MB/sec execute 545 sec latency 48.469 ms 2024-04-02T04:53:55.644 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:55 smithi122 ceph-mon[53211]: pgmap v260: 97 pgs: 3 remapped+peering, 1 peering, 93 active+clean; 330 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 2.5 MiB/s wr, 327 op/s; 1.3 MiB/s, 54 keys/s, 37 objects/s recovering 2024-04-02T04:53:56.339 INFO:teuthology.orchestra.run.smithi053.stdout: 5 265847 9.61 MB/sec execute 546 sec latency 42.297 ms 2024-04-02T04:53:56.373 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:56 smithi053 ceph-mon[67180]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 55/1497 objects degraded (3.674%), 6 pgs degraded) 2024-04-02T04:53:56.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:56 smithi122 ceph-mon[53211]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 55/1497 objects degraded (3.674%), 6 pgs degraded) 2024-04-02T04:53:57.340 INFO:teuthology.orchestra.run.smithi053.stdout: 5 266195 9.61 MB/sec execute 547 sec latency 52.867 ms 2024-04-02T04:53:57.400 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:57 smithi053 ceph-mon[67180]: pgmap v261: 97 pgs: 1 peering, 96 active+clean; 326 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.3 KiB/s rd, 1.9 MiB/s wr, 251 op/s; 2.2 MiB/s, 38 keys/s, 31 objects/s recovering 2024-04-02T04:53:57.400 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:57 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:57.400 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:57 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:53:57.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:57 smithi122 ceph-mon[53211]: pgmap v261: 97 pgs: 1 peering, 96 active+clean; 326 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.3 KiB/s rd, 1.9 MiB/s wr, 251 op/s; 2.2 MiB/s, 38 keys/s, 31 objects/s recovering 2024-04-02T04:53:57.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:57 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:53:57.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:57 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:53:58.339 INFO:teuthology.orchestra.run.smithi053.stdout: 5 266514 9.62 MB/sec execute 548 sec latency 51.811 ms 2024-04-02T04:53:58.662 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:53:58 smithi053 ceph-mon[67180]: pgmap v262: 97 pgs: 97 active+clean; 333 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.1 KiB/s rd, 3.6 MiB/s wr, 431 op/s; 1.7 MiB/s, 29 keys/s, 24 objects/s recovering 2024-04-02T04:53:58.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:53:58 smithi122 ceph-mon[53211]: pgmap v262: 97 pgs: 97 active+clean; 333 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.1 KiB/s rd, 3.6 MiB/s wr, 431 op/s; 1.7 MiB/s, 29 keys/s, 24 objects/s recovering 2024-04-02T04:53:59.340 INFO:teuthology.orchestra.run.smithi053.stdout: 5 266778 9.61 MB/sec execute 549 sec latency 52.672 ms 2024-04-02T04:54:00.340 INFO:teuthology.orchestra.run.smithi053.stdout: 5 267094 9.60 MB/sec execute 550 sec latency 50.163 ms 2024-04-02T04:54:00.681 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:00 smithi053 ceph-mon[67180]: pgmap v263: 97 pgs: 97 active+clean; 338 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.3 MiB/s wr, 421 op/s; 1.5 MiB/s, 26 keys/s, 21 objects/s recovering 2024-04-02T04:54:00.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:00 smithi122 ceph-mon[53211]: pgmap v263: 97 pgs: 97 active+clean; 338 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.3 MiB/s wr, 421 op/s; 1.5 MiB/s, 26 keys/s, 21 objects/s recovering 2024-04-02T04:54:01.340 INFO:teuthology.orchestra.run.smithi053.stdout: 5 267386 9.59 MB/sec execute 551 sec latency 46.144 ms 2024-04-02T04:54:01.972 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-02T04:54:02.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-02T04:54:02.339 INFO:teuthology.orchestra.run.smithi053.stdout: 5 267664 9.59 MB/sec execute 552 sec latency 62.672 ms 2024-04-02T04:54:02.992 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:02 smithi053 ceph-mon[67180]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-02T04:54:02.992 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:02 smithi053 ceph-mon[67180]: Upgrade: osd.7 is safe to restart 2024-04-02T04:54:02.992 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:02 smithi053 ceph-mon[67180]: pgmap v264: 97 pgs: 97 active+clean; 338 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.8 MiB/s wr, 369 op/s; 1.3 MiB/s, 23 keys/s, 15 objects/s recovering 2024-04-02T04:54:03.144 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:02 smithi122 ceph-mon[53211]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-02T04:54:03.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:02 smithi122 ceph-mon[53211]: Upgrade: osd.7 is safe to restart 2024-04-02T04:54:03.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:02 smithi122 ceph-mon[53211]: pgmap v264: 97 pgs: 97 active+clean; 338 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.8 MiB/s wr, 369 op/s; 1.3 MiB/s, 23 keys/s, 15 objects/s recovering 2024-04-02T04:54:03.340 INFO:teuthology.orchestra.run.smithi053.stdout: 5 268081 9.60 MB/sec execute 553 sec latency 36.144 ms 2024-04-02T04:54:04.340 INFO:teuthology.orchestra.run.smithi053.stdout: 5 268514 9.59 MB/sec execute 554 sec latency 42.060 ms 2024-04-02T04:54:05.072 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:04 smithi053 ceph-mon[67180]: Upgrade: Updating osd.7 2024-04-02T04:54:05.072 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:04 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:05.072 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:04 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-02T04:54:05.072 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:04 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:54:05.072 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:04 smithi053 ceph-mon[67180]: Deploying daemon osd.7 on smithi122 2024-04-02T04:54:05.072 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:04 smithi053 ceph-mon[67180]: pgmap v265: 97 pgs: 97 active+clean; 335 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.4 MiB/s wr, 411 op/s; 1.1 MiB/s, 7 objects/s recovering 2024-04-02T04:54:05.340 INFO:teuthology.orchestra.run.smithi053.stdout: 5 268802 9.58 MB/sec execute 555 sec latency 46.610 ms 2024-04-02T04:54:05.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:04 smithi122 ceph-mon[53211]: Upgrade: Updating osd.7 2024-04-02T04:54:05.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:04 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:05.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:04 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-02T04:54:05.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:04 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:54:05.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:04 smithi122 ceph-mon[53211]: Deploying daemon osd.7 on smithi122 2024-04-02T04:54:05.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:04 smithi122 ceph-mon[53211]: pgmap v265: 97 pgs: 97 active+clean; 335 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.4 MiB/s wr, 411 op/s; 1.1 MiB/s, 7 objects/s recovering 2024-04-02T04:54:06.340 INFO:teuthology.orchestra.run.smithi053.stdout: 5 269104 9.58 MB/sec execute 556 sec latency 49.807 ms 2024-04-02T04:54:06.855 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:06 smithi053 ceph-mon[67180]: pgmap v266: 97 pgs: 97 active+clean; 332 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 4.5 MiB/s wr, 479 op/s; 641 KiB/s, 2 objects/s recovering 2024-04-02T04:54:06.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:06 smithi122 ceph-mon[53211]: pgmap v266: 97 pgs: 97 active+clean; 332 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 4.5 MiB/s wr, 479 op/s; 641 KiB/s, 2 objects/s recovering 2024-04-02T04:54:07.340 INFO:teuthology.orchestra.run.smithi053.stdout: 5 269441 9.59 MB/sec execute 557 sec latency 48.049 ms 2024-04-02T04:54:07.870 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:07 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:08.097 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:07 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:08.340 INFO:teuthology.orchestra.run.smithi053.stdout: 5 269787 9.59 MB/sec execute 558 sec latency 35.539 ms 2024-04-02T04:54:08.395 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:08 smithi122 systemd[1]: Stopping Ceph osd.7 for 48e16742-f0aa-11ee-b647-cb9ed24678a4... 2024-04-02T04:54:08.779 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:08 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[40446]: 2024-04-02T04:54:08.506+0000 7f64d8a4f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T04:54:08.779 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:08 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[40446]: 2024-04-02T04:54:08.506+0000 7f64d8a4f700 -1 osd.7 138 *** Got signal Terminated *** 2024-04-02T04:54:08.779 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:08 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[40446]: 2024-04-02T04:54:08.506+0000 7f64d8a4f700 -1 osd.7 138 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T04:54:09.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:08 smithi053 ceph-mon[67180]: osd.7 marked itself down and dead 2024-04-02T04:54:09.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:08 smithi053 ceph-mon[67180]: pgmap v267: 97 pgs: 97 active+clean; 335 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.7 MiB/s wr, 574 op/s 2024-04-02T04:54:09.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:08 smithi122 ceph-mon[53211]: osd.7 marked itself down and dead 2024-04-02T04:54:09.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:08 smithi122 ceph-mon[53211]: pgmap v267: 97 pgs: 97 active+clean; 335 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.7 MiB/s wr, 574 op/s 2024-04-02T04:54:09.340 INFO:teuthology.orchestra.run.smithi053.stdout: 5 269992 9.60 MB/sec execute 559 sec latency 569.154 ms 2024-04-02T04:54:09.761 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:09 smithi122 podman[64452]: 2024-04-02 04:54:09.454673612 +0000 UTC m=+1.168986830 container died 9b3cc398254d4c8b68c99dea3b8038cf18b773490520489c58c369f2be98b100 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, distribution-scope=public, vcs-type=git, ceph=True, release=754, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, version=8, RELEASE=HEAD, io.buildah.version=1.19.8, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-02T04:54:10.032 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:09 smithi053 ceph-mon[67180]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-02T04:54:10.032 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:09 smithi053 ceph-mon[67180]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-02T04:54:10.032 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:09 smithi053 ceph-mon[67180]: osdmap e139: 8 total, 7 up, 8 in 2024-04-02T04:54:10.112 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:09 smithi122 ceph-mon[53211]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-02T04:54:10.112 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:09 smithi122 ceph-mon[53211]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-02T04:54:10.112 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:09 smithi122 ceph-mon[53211]: osdmap e139: 8 total, 7 up, 8 in 2024-04-02T04:54:10.112 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:09 smithi122 podman[64452]: 2024-04-02 04:54:09.840431586 +0000 UTC m=+1.554744752 container cleanup 9b3cc398254d4c8b68c99dea3b8038cf18b773490520489c58c369f2be98b100 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7, GIT_CLEAN=True, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , version=8, GIT_BRANCH=HEAD, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, vendor=Red Hat, Inc., io.openshift.expose-services=, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-02T04:54:10.112 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:09 smithi122 bash[64452]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7 2024-04-02T04:54:10.340 INFO:teuthology.orchestra.run.smithi053.stdout: 5 270144 9.59 MB/sec execute 560 sec latency 1045.051 ms 2024-04-02T04:54:10.395 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:10 smithi122 podman[64469]: 2024-04-02 04:54:10.108914584 +0000 UTC m=+0.652708757 container remove 9b3cc398254d4c8b68c99dea3b8038cf18b773490520489c58c369f2be98b100 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7, io.openshift.expose-services=, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, io.openshift.tags=base centos centos-stream, architecture=x86_64, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD) 2024-04-02T04:54:10.705 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:10 smithi122 podman[64588]: 2024-04-02 04:54:10.350622783 +0000 UTC m=+0.018439096 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:54:10.705 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:10 smithi122 podman[64588]: 2024-04-02 04:54:10.548375526 +0000 UTC m=+0.216191811 container create 2cc683bb5d6da10889def5c0d13970dee908036d96ddb525cd0e7a1e93f75fd9 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240326, RELEASE=squid-6cec793, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-02T04:54:10.795 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:10 smithi053 ceph-mon[67180]: osdmap e140: 8 total, 7 up, 8 in 2024-04-02T04:54:10.795 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:10 smithi053 ceph-mon[67180]: pgmap v270: 97 pgs: 2 active+undersized, 6 active+undersized+degraded, 9 peering, 7 stale+active+clean, 73 active+clean; 337 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.1 MiB/s wr, 616 op/s; 42/1599 objects degraded (2.627%) 2024-04-02T04:54:11.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:10 smithi122 ceph-mon[53211]: osdmap e140: 8 total, 7 up, 8 in 2024-04-02T04:54:11.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:10 smithi122 ceph-mon[53211]: pgmap v270: 97 pgs: 2 active+undersized, 6 active+undersized+degraded, 9 peering, 7 stale+active+clean, 73 active+clean; 337 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.1 MiB/s wr, 616 op/s; 42/1599 objects degraded (2.627%) 2024-04-02T04:54:11.145 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:10 smithi122 podman[64588]: 2024-04-02 04:54:10.805016986 +0000 UTC m=+0.472833295 container init 2cc683bb5d6da10889def5c0d13970dee908036d96ddb525cd0e7a1e93f75fd9 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7-deactivate, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-6cec793, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.license=GPLv2) 2024-04-02T04:54:11.145 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:10 smithi122 podman[64588]: 2024-04-02 04:54:10.809259424 +0000 UTC m=+0.477075718 container start 2cc683bb5d6da10889def5c0d13970dee908036d96ddb525cd0e7a1e93f75fd9 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-6cec793, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-02T04:54:11.145 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:10 smithi122 podman[64588]: 2024-04-02 04:54:10.912984238 +0000 UTC m=+0.580800538 container attach 2cc683bb5d6da10889def5c0d13970dee908036d96ddb525cd0e7a1e93f75fd9 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-6cec793, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-02T04:54:11.340 INFO:teuthology.orchestra.run.smithi053.stdout: 5 270459 9.58 MB/sec execute 561 sec latency 32.570 ms 2024-04-02T04:54:11.645 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:11 smithi122 podman[64588]: 2024-04-02 04:54:11.185014363 +0000 UTC m=+0.852830668 container died 2cc683bb5d6da10889def5c0d13970dee908036d96ddb525cd0e7a1e93f75fd9 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-6cec793, ceph=True, org.label-schema.build-date=20240326, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-02T04:54:11.827 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:11 smithi053 ceph-mon[67180]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-02T04:54:11.827 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:11 smithi053 ceph-mon[67180]: Health check failed: Degraded data redundancy: 42/1599 objects degraded (2.627%), 6 pgs degraded (PG_DEGRADED) 2024-04-02T04:54:11.827 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:11 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:11.827 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:11 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:54:12.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:11 smithi122 ceph-mon[53211]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-02T04:54:12.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:11 smithi122 ceph-mon[53211]: Health check failed: Degraded data redundancy: 42/1599 objects degraded (2.627%), 6 pgs degraded (PG_DEGRADED) 2024-04-02T04:54:12.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:11 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:12.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:11 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:54:12.145 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:11 smithi122 podman[64614]: 2024-04-02 04:54:11.893281647 +0000 UTC m=+0.701608318 container remove 2cc683bb5d6da10889def5c0d13970dee908036d96ddb525cd0e7a1e93f75fd9 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-6cec793, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True) 2024-04-02T04:54:12.145 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:11 smithi122 systemd[1]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.7.service: Deactivated successfully. 2024-04-02T04:54:12.145 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:11 smithi122 systemd[1]: Stopped Ceph osd.7 for 48e16742-f0aa-11ee-b647-cb9ed24678a4. 2024-04-02T04:54:12.145 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:11 smithi122 systemd[1]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.7.service: Consumed 7min 808ms CPU time. 2024-04-02T04:54:12.340 INFO:teuthology.orchestra.run.smithi053.stdout: 5 270818 9.57 MB/sec execute 562 sec latency 45.145 ms 2024-04-02T04:54:12.439 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:12 smithi122 systemd[1]: Starting Ceph osd.7 for 48e16742-f0aa-11ee-b647-cb9ed24678a4... 2024-04-02T04:54:12.439 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:12 smithi122 podman[64709]: 2024-04-02 04:54:12.309654954 +0000 UTC m=+0.023265256 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:54:12.700 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:12 smithi122 podman[64709]: 2024-04-02 04:54:12.436041886 +0000 UTC m=+0.149652236 container create e0b6169c14233256450548d4501dd6cbf77c5f3a9f7d50878f83672b8f21a953 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7-activate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-6cec793, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T04:54:12.852 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:12 smithi053 ceph-mon[67180]: pgmap v271: 97 pgs: 2 active+undersized, 12 active+undersized+degraded, 9 peering, 6 stale+active+clean, 68 active+clean; 340 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.8 MiB/s wr, 416 op/s; 68/1608 objects degraded (4.229%) 2024-04-02T04:54:13.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:12 smithi122 ceph-mon[53211]: pgmap v271: 97 pgs: 2 active+undersized, 12 active+undersized+degraded, 9 peering, 6 stale+active+clean, 68 active+clean; 340 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.8 MiB/s wr, 416 op/s; 68/1608 objects degraded (4.229%) 2024-04-02T04:54:13.145 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:12 smithi122 podman[64709]: 2024-04-02 04:54:12.697615243 +0000 UTC m=+0.411225534 container init e0b6169c14233256450548d4501dd6cbf77c5f3a9f7d50878f83672b8f21a953 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-6cec793, org.label-schema.vendor=CentOS) 2024-04-02T04:54:13.145 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:12 smithi122 podman[64709]: 2024-04-02 04:54:12.702528945 +0000 UTC m=+0.416139241 container start e0b6169c14233256450548d4501dd6cbf77c5f3a9f7d50878f83672b8f21a953 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-6cec793, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-02T04:54:13.145 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:12 smithi122 podman[64709]: 2024-04-02 04:54:12.797473585 +0000 UTC m=+0.511083876 container attach e0b6169c14233256450548d4501dd6cbf77c5f3a9f7d50878f83672b8f21a953 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7-activate, RELEASE=squid-6cec793, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD) 2024-04-02T04:54:13.340 INFO:teuthology.orchestra.run.smithi053.stdout: 5 271099 9.56 MB/sec execute 563 sec latency 40.345 ms 2024-04-02T04:54:13.622 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:13 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7-activate[64724]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-02T04:54:13.622 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:13 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7-activate[64724]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-02T04:54:13.623 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:13 smithi122 bash[64709]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-02T04:54:13.623 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:13 smithi122 bash[64709]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-02T04:54:13.623 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:13 smithi122 bash[64709]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-02T04:54:13.623 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:13 smithi122 bash[64709]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-02T04:54:13.623 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:13 smithi122 bash[64709]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-04-02T04:54:13.623 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:13 smithi122 bash[64709]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-02T04:54:13.623 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:13 smithi122 bash[64709]: --> ceph-volume raw activate successful for osd ID: 7 2024-04-02T04:54:13.623 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:13 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7-activate[64724]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-02T04:54:13.623 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:13 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7-activate[64724]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-02T04:54:13.623 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:13 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7-activate[64724]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-04-02T04:54:13.623 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:13 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7-activate[64724]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-02T04:54:13.623 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:13 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7-activate[64724]: --> ceph-volume raw activate successful for osd ID: 7 2024-04-02T04:54:13.623 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:13 smithi122 podman[64709]: 2024-04-02 04:54:13.310282329 +0000 UTC m=+1.023892622 container died e0b6169c14233256450548d4501dd6cbf77c5f3a9f7d50878f83672b8f21a953 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-6cec793, org.label-schema.vendor=CentOS) 2024-04-02T04:54:14.340 INFO:teuthology.orchestra.run.smithi053.stdout: 5 271534 9.58 MB/sec execute 564 sec latency 52.026 ms 2024-04-02T04:54:14.395 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:14 smithi122 podman[64846]: 2024-04-02 04:54:14.027790163 +0000 UTC m=+0.711617882 container remove e0b6169c14233256450548d4501dd6cbf77c5f3a9f7d50878f83672b8f21a953 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7-activate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-6cec793, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-02T04:54:14.395 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:14 smithi122 podman[64906]: 2024-04-02 04:54:14.154070884 +0000 UTC m=+0.019575550 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:54:14.395 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:14 smithi122 podman[64906]: 2024-04-02 04:54:14.270334368 +0000 UTC m=+0.135838999 container create 86ee236f94268657c18e051249afa168e95cce7a09ae51108098f60e0fd9ab82 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-6cec793) 2024-04-02T04:54:14.650 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:14 smithi053 ceph-mon[67180]: pgmap v272: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 340 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.9 KiB/s rd, 5.5 MiB/s wr, 415 op/s; 242/1527 objects degraded (15.848%) 2024-04-02T04:54:14.650 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:14 smithi053 ceph-mon[67180]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-04-02T04:54:14.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:14 smithi122 ceph-mon[53211]: pgmap v272: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 340 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.9 KiB/s rd, 5.5 MiB/s wr, 415 op/s; 242/1527 objects degraded (15.848%) 2024-04-02T04:54:14.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:14 smithi122 ceph-mon[53211]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-04-02T04:54:14.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:14 smithi122 podman[64906]: 2024-04-02 04:54:14.506302177 +0000 UTC m=+0.371806808 container init 86ee236f94268657c18e051249afa168e95cce7a09ae51108098f60e0fd9ab82 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-6cec793, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326) 2024-04-02T04:54:14.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:14 smithi122 podman[64906]: 2024-04-02 04:54:14.509538701 +0000 UTC m=+0.375043341 container start 86ee236f94268657c18e051249afa168e95cce7a09ae51108098f60e0fd9ab82 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-6cec793, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-02T04:54:14.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:14 smithi122 ceph-osd[64925]: -- 172.21.15.122:0/3083254176 <== mon.1 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55609c657680 con 0x55609c621000 2024-04-02T04:54:14.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:14 smithi122 bash[64906]: 86ee236f94268657c18e051249afa168e95cce7a09ae51108098f60e0fd9ab82 2024-04-02T04:54:14.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:14 smithi122 systemd[1]: Started Ceph osd.7 for 48e16742-f0aa-11ee-b647-cb9ed24678a4. 2024-04-02T04:54:15.340 INFO:teuthology.orchestra.run.smithi053.stdout: 5 272015 9.57 MB/sec execute 565 sec latency 37.656 ms 2024-04-02T04:54:15.395 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:15 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:54:15.055+0000 7f892ccdd7c0 -1 Falling back to public interface 2024-04-02T04:54:15.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:15 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:15.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:15 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:15.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:15 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:54:15.909 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:15 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:15.909 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:15 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:15.909 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:15 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:54:16.340 INFO:teuthology.orchestra.run.smithi053.stdout: 5 272326 9.56 MB/sec execute 566 sec latency 73.148 ms 2024-04-02T04:54:16.716 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:16 smithi053 ceph-mon[67180]: pgmap v273: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 334 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.7 MiB/s wr, 280 op/s; 241/1527 objects degraded (15.783%) 2024-04-02T04:54:17.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:16 smithi122 ceph-mon[53211]: pgmap v273: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 334 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.7 MiB/s wr, 280 op/s; 241/1527 objects degraded (15.783%) 2024-04-02T04:54:17.340 INFO:teuthology.orchestra.run.smithi053.stdout: 5 272623 9.56 MB/sec execute 567 sec latency 57.370 ms 2024-04-02T04:54:17.755 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:17 smithi053 ceph-mon[67180]: Health check update: Degraded data redundancy: 241/1527 objects degraded (15.783%), 35 pgs degraded (PG_DEGRADED) 2024-04-02T04:54:18.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:17 smithi122 ceph-mon[53211]: Health check update: Degraded data redundancy: 241/1527 objects degraded (15.783%), 35 pgs degraded (PG_DEGRADED) 2024-04-02T04:54:18.340 INFO:teuthology.orchestra.run.smithi053.stdout: 5 273079 9.58 MB/sec execute 568 sec latency 41.266 ms 2024-04-02T04:54:18.761 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:18 smithi053 ceph-mon[67180]: pgmap v274: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 337 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.8 MiB/s wr, 371 op/s; 239/1575 objects degraded (15.175%) 2024-04-02T04:54:19.128 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:18 smithi122 ceph-mon[53211]: pgmap v274: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 337 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.8 MiB/s wr, 371 op/s; 239/1575 objects degraded (15.175%) 2024-04-02T04:54:19.340 INFO:teuthology.orchestra.run.smithi053.stdout: 5 273526 9.60 MB/sec execute 569 sec latency 29.948 ms 2024-04-02T04:54:20.340 INFO:teuthology.orchestra.run.smithi053.stdout: 5 273832 9.59 MB/sec execute 570 sec latency 49.605 ms 2024-04-02T04:54:20.789 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:20 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:20.789 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:20 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:20.789 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:20 smithi053 ceph-mon[67180]: pgmap v275: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 342 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.6 KiB/s rd, 5.0 MiB/s wr, 487 op/s; 242/1641 objects degraded (14.747%) 2024-04-02T04:54:20.894 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:20 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:20.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:20 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:20.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:20 smithi122 ceph-mon[53211]: pgmap v275: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 342 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.6 KiB/s rd, 5.0 MiB/s wr, 487 op/s; 242/1641 objects degraded (14.747%) 2024-04-02T04:54:21.259 INFO:teuthology.orchestra.run.smithi053.stdout:true 2024-04-02T04:54:21.340 INFO:teuthology.orchestra.run.smithi053.stdout: 5 274068 9.58 MB/sec execute 571 sec latency 56.494 ms 2024-04-02T04:54:22.055 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:21 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:22.055 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:21 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:22.055 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:21 smithi053 ceph-mon[67180]: from='client.34406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:54:22.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:21 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:22.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:21 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:22.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:21 smithi122 ceph-mon[53211]: from='client.34406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:54:22.316 INFO:teuthology.orchestra.run.smithi053.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-02T04:54:22.316 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager.smithi053 smithi053 *:9093,9094 running (8m) 2m ago 17m 23.6M - 0.23.0 ba2b418f427c bf50c1cd2b3d 2024-04-02T04:54:22.316 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi053 smithi053 running (5m) 2m ago 17m 6660k - 19.0.0-1979-g6cec793d 78f830190839 4ce8915e0157 2024-04-02T04:54:22.316 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi122 smithi122 running (5m) 2s ago 17m 6660k - 19.0.0-1979-g6cec793d 78f830190839 e2b5369e6fdf 2024-04-02T04:54:22.316 INFO:teuthology.orchestra.run.smithi053.stdout:grafana.smithi053 smithi053 *:3000 error 2m ago 17m - - 2024-04-02T04:54:22.317 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi053.nfrfot smithi053 running (12m) 2m ago 12m 37.4M - 17.2.0 e1d6a67b021e c94e9e643514 2024-04-02T04:54:22.317 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi122.mwoyoh smithi122 running (12m) 2s ago 12m 16.5M - 17.2.0 e1d6a67b021e 73376d9ee4b8 2024-04-02T04:54:22.317 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi053.legdmv smithi053 *:8443,9283,8765 running (7m) 2m ago 19m 506M - 19.0.0-1979-g6cec793d 78f830190839 c12d516a31bc 2024-04-02T04:54:22.317 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi122.cpwlyk smithi122 *:8443,9283,8765 running (6m) 2s ago 17m 452M - 19.0.0-1979-g6cec793d 78f830190839 9a545c5e7d81 2024-04-02T04:54:22.317 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi053 smithi053 running (6m) 2m ago 19m 50.4M 2048M 19.0.0-1979-g6cec793d 78f830190839 f53a44f19c00 2024-04-02T04:54:22.317 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi122 smithi122 running (6m) 2s ago 17m 42.8M 2048M 19.0.0-1979-g6cec793d 78f830190839 44b200fed3c5 2024-04-02T04:54:22.317 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.0.0.smithi053.mhkdcb smithi053 *:2049 running (11m) 2m ago 11m 296M - 4.0 e1d6a67b021e 07e5da7bd2c5 2024-04-02T04:54:22.317 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.1.0.smithi122.mauvsb smithi122 *:2049 running (11m) 2s ago 11m 84.3M - 4.0 e1d6a67b021e 4f866685005e 2024-04-02T04:54:22.317 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi053 smithi053 *:9100 running (8m) 2m ago 17m 18.4M - 1.3.1 1dbe0e931976 773b71971e67 2024-04-02T04:54:22.317 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi122 smithi122 *:9100 running (8m) 2s ago 17m 16.3M - 1.3.1 1dbe0e931976 475d439c7980 2024-04-02T04:54:22.317 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 smithi053 running (4m) 2m ago 16m 334M 1937M 19.0.0-1979-g6cec793d 78f830190839 490c346c7639 2024-04-02T04:54:22.317 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 smithi053 running (4m) 2m ago 15m 283M 1937M 19.0.0-1979-g6cec793d 78f830190839 e70e40949c2f 2024-04-02T04:54:22.317 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 smithi053 running (3m) 2m ago 15m 200M 1937M 19.0.0-1979-g6cec793d 78f830190839 4f053827e0dd 2024-04-02T04:54:22.317 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 smithi053 running (2m) 2m ago 15m 12.7M 1937M 19.0.0-1979-g6cec793d 78f830190839 5cd6de7a6535 2024-04-02T04:54:22.317 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 smithi122 running (118s) 2s ago 14m 328M 2704M 19.0.0-1979-g6cec793d 78f830190839 40d6ab1aac4b 2024-04-02T04:54:22.318 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 smithi122 running (82s) 2s ago 14m 311M 2704M 19.0.0-1979-g6cec793d 78f830190839 be26f425f51b 2024-04-02T04:54:22.318 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 smithi122 running (46s) 2s ago 13m 262M 2704M 19.0.0-1979-g6cec793d 78f830190839 aab18af876da 2024-04-02T04:54:22.318 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 smithi122 running (8s) 2s ago 13m 12.4M 2704M 19.0.0-1979-g6cec793d 78f830190839 86ee236f9426 2024-04-02T04:54:22.318 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus.smithi053 smithi053 *:9095 running (7m) 2m ago 16m 69.6M - 2.33.4 514e6a882f6e 1dd55399da10 2024-04-02T04:54:22.340 INFO:teuthology.orchestra.run.smithi053.stdout: 5 274366 9.57 MB/sec execute 572 sec latency 46.803 ms 2024-04-02T04:54:22.953 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:54:22.953 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": { 2024-04-02T04:54:22.953 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:54:22.953 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:54:22.953 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": { 2024-04-02T04:54:22.953 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:54:22.953 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:54:22.953 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": { 2024-04-02T04:54:22.953 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 7 2024-04-02T04:54:22.953 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:54:22.954 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": { 2024-04-02T04:54:22.954 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-02T04:54:22.954 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:54:22.954 INFO:teuthology.orchestra.run.smithi053.stdout: "overall": { 2024-04-02T04:54:22.954 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-04-02T04:54:22.954 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 11 2024-04-02T04:54:22.954 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-02T04:54:22.954 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:54:23.060 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:22 smithi053 ceph-mon[67180]: from='client.34410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:54:23.060 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:22 smithi053 ceph-mon[67180]: from='client.34414 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:54:23.060 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:22 smithi053 ceph-mon[67180]: pgmap v276: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 343 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 4.4 MiB/s wr, 433 op/s; 242/1656 objects degraded (14.614%) 2024-04-02T04:54:23.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:22 smithi122 ceph-mon[53211]: from='client.34410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:54:23.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:22 smithi122 ceph-mon[53211]: from='client.34414 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:54:23.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:22 smithi122 ceph-mon[53211]: pgmap v276: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 343 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 4.4 MiB/s wr, 433 op/s; 242/1656 objects degraded (14.614%) 2024-04-02T04:54:23.341 INFO:teuthology.orchestra.run.smithi053.stdout: 5 274613 9.56 MB/sec execute 573 sec latency 60.683 ms 2024-04-02T04:54:23.505 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:54:23.505 INFO:teuthology.orchestra.run.smithi053.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae", 2024-04-02T04:54:23.505 INFO:teuthology.orchestra.run.smithi053.stdout: "in_progress": true, 2024-04-02T04:54:23.506 INFO:teuthology.orchestra.run.smithi053.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-02T04:54:23.506 INFO:teuthology.orchestra.run.smithi053.stdout: "services_complete": [ 2024-04-02T04:54:23.506 INFO:teuthology.orchestra.run.smithi053.stdout: "osd", 2024-04-02T04:54:23.506 INFO:teuthology.orchestra.run.smithi053.stdout: "mon", 2024-04-02T04:54:23.506 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr", 2024-04-02T04:54:23.506 INFO:teuthology.orchestra.run.smithi053.stdout: "crash" 2024-04-02T04:54:23.507 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-04-02T04:54:23.507 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": "14/23 daemons upgraded", 2024-04-02T04:54:23.507 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "Currently upgrading osd daemons", 2024-04-02T04:54:23.507 INFO:teuthology.orchestra.run.smithi053.stdout: "is_paused": false 2024-04-02T04:54:23.507 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:54:24.075 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:23 smithi053 ceph-mon[67180]: Health check update: Degraded data redundancy: 242/1656 objects degraded (14.614%), 35 pgs degraded (PG_DEGRADED) 2024-04-02T04:54:24.075 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:23 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/2939100169' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:54:24.075 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:23 smithi053 ceph-mon[67180]: from='client.44315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:54:24.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:23 smithi122 ceph-mon[53211]: Health check update: Degraded data redundancy: 242/1656 objects degraded (14.614%), 35 pgs degraded (PG_DEGRADED) 2024-04-02T04:54:24.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:23 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/2939100169' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:54:24.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:23 smithi122 ceph-mon[53211]: from='client.44315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:54:24.163 INFO:teuthology.orchestra.run.smithi053.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; all OSDs are running squid or later but require_osd_release < squid; Degraded data redundancy: 242/1656 objects degraded (14.614%), 35 pgs degraded 2024-04-02T04:54:24.163 INFO:teuthology.orchestra.run.smithi053.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-02T04:54:24.163 INFO:teuthology.orchestra.run.smithi053.stdout: daemon grafana.smithi053 on smithi053 is in error state 2024-04-02T04:54:24.163 INFO:teuthology.orchestra.run.smithi053.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-02T04:54:24.163 INFO:teuthology.orchestra.run.smithi053.stdout: osd.7 (root=default,host=smithi122) is down 2024-04-02T04:54:24.163 INFO:teuthology.orchestra.run.smithi053.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-04-02T04:54:24.163 INFO:teuthology.orchestra.run.smithi053.stdout: all OSDs are running squid or later but require_osd_release < squid 2024-04-02T04:54:24.164 INFO:teuthology.orchestra.run.smithi053.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 242/1656 objects degraded (14.614%), 35 pgs degraded 2024-04-02T04:54:24.164 INFO:teuthology.orchestra.run.smithi053.stdout: pg 1.0 is active+undersized+degraded, acting [0,6] 2024-04-02T04:54:24.164 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.0 is active+undersized+degraded, acting [1,0] 2024-04-02T04:54:24.164 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.3 is active+undersized+degraded, acting [5,2] 2024-04-02T04:54:24.164 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.4 is active+undersized+degraded, acting [1,0] 2024-04-02T04:54:24.164 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.5 is active+undersized+degraded, acting [0,4] 2024-04-02T04:54:24.164 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.7 is active+undersized+degraded, acting [6,2] 2024-04-02T04:54:24.164 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.8 is active+undersized+degraded, acting [0,1] 2024-04-02T04:54:24.164 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.9 is active+undersized+degraded, acting [1,3] 2024-04-02T04:54:24.164 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.a is active+undersized+degraded, acting [1,3] 2024-04-02T04:54:24.164 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.e is active+undersized+degraded, acting [2,3] 2024-04-02T04:54:24.164 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.f is active+undersized+degraded, acting [4,0] 2024-04-02T04:54:24.164 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.12 is active+undersized+degraded, acting [5,3] 2024-04-02T04:54:24.164 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.18 is active+undersized+degraded, acting [5,4] 2024-04-02T04:54:24.164 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.1a is active+undersized+degraded, acting [6,4] 2024-04-02T04:54:24.164 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.1b is active+undersized+degraded, acting [3,6] 2024-04-02T04:54:24.165 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.1c is active+undersized+degraded, acting [5,2] 2024-04-02T04:54:24.165 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.1d is active+undersized+degraded, acting [6,0] 2024-04-02T04:54:24.165 INFO:teuthology.orchestra.run.smithi053.stdout: pg 2.1f is active+undersized+degraded, acting [0,4] 2024-04-02T04:54:24.165 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.7 is active+undersized+degraded, acting [3,0] 2024-04-02T04:54:24.165 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.8 is active+undersized+degraded, acting [3,1] 2024-04-02T04:54:24.165 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.9 is active+undersized+degraded, acting [4,2] 2024-04-02T04:54:24.165 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.d is active+undersized+degraded, acting [5,6] 2024-04-02T04:54:24.165 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.e is active+undersized+degraded, acting [4,1] 2024-04-02T04:54:24.165 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.f is active+undersized+degraded, acting [4,0] 2024-04-02T04:54:24.165 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.11 is active+undersized+degraded, acting [4,6] 2024-04-02T04:54:24.165 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.12 is active+undersized+degraded, acting [0,3] 2024-04-02T04:54:24.165 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.13 is active+undersized+degraded, acting [4,2] 2024-04-02T04:54:24.165 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.14 is active+undersized+degraded, acting [4,6] 2024-04-02T04:54:24.165 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.15 is active+undersized+degraded, acting [3,4] 2024-04-02T04:54:24.165 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.16 is active+undersized+degraded, acting [5,1] 2024-04-02T04:54:24.166 INFO:teuthology.orchestra.run.smithi053.stdout: pg 3.1b is active+undersized+degraded, acting [0,4] 2024-04-02T04:54:24.166 INFO:teuthology.orchestra.run.smithi053.stdout: pg 4.0 is active+undersized+degraded, acting [3,0] 2024-04-02T04:54:24.166 INFO:teuthology.orchestra.run.smithi053.stdout: pg 4.5 is active+undersized+degraded, acting [6,4] 2024-04-02T04:54:24.166 INFO:teuthology.orchestra.run.smithi053.stdout: pg 4.a is active+undersized+degraded, acting [6,2] 2024-04-02T04:54:24.166 INFO:teuthology.orchestra.run.smithi053.stdout: pg 4.14 is active+undersized+degraded, acting [3,1] 2024-04-02T04:54:24.340 INFO:teuthology.orchestra.run.smithi053.stdout: 5 274898 9.56 MB/sec execute 574 sec latency 43.522 ms 2024-04-02T04:54:24.645 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:24 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:54:24.551+0000 7f892ccdd7c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-04-02T04:54:25.086 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:24 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/4264770172' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:54:25.086 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:24 smithi053 ceph-mon[67180]: pgmap v277: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 339 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.8 MiB/s wr, 560 op/s; 236/1602 objects degraded (14.732%) 2024-04-02T04:54:25.086 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:24 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:25.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:24 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/4264770172' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:54:25.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:24 smithi122 ceph-mon[53211]: pgmap v277: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 339 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.8 MiB/s wr, 560 op/s; 236/1602 objects degraded (14.732%) 2024-04-02T04:54:25.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:24 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:25.341 INFO:teuthology.orchestra.run.smithi053.stdout: 5 275276 9.57 MB/sec execute 575 sec latency 45.298 ms 2024-04-02T04:54:26.103 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:26.103 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:54:26.104 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:54:26.104 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:25 smithi053 ceph-mon[67180]: pgmap v278: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 339 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.0 MiB/s wr, 539 op/s; 236/1602 objects degraded (14.732%) 2024-04-02T04:54:26.104 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:26.104 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:26.104 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:54:26.104 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:54:26.104 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:54:26.104 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:54:26.104 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:54:26.104 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:25 smithi053 ceph-mon[67180]: Upgrade: Setting container_image for all osd 2024-04-02T04:54:26.104 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:26.104 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-02T04:54:26.104 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-02T04:54:26.104 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-02T04:54:26.104 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-02T04:54:26.105 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-02T04:54:26.105 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-02T04:54:26.105 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-02T04:54:26.105 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-02T04:54:26.105 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-02T04:54:26.105 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-02T04:54:26.109 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-02T04:54:26.109 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-02T04:54:26.109 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-04-02T04:54:26.109 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-04-02T04:54:26.109 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-04-02T04:54:26.110 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-04-02T04:54:26.110 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:25 smithi053 ceph-mon[67180]: Upgrade: Setting require_osd_release to 19 squid 2024-04-02T04:54:26.110 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:25 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-02T04:54:26.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:26.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:54:26.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:54:26.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:25 smithi122 ceph-mon[53211]: pgmap v278: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 339 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.0 MiB/s wr, 539 op/s; 236/1602 objects degraded (14.732%) 2024-04-02T04:54:26.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:26.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:26.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:54:26.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:54:26.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:54:26.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:54:26.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:54:26.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:25 smithi122 ceph-mon[53211]: Upgrade: Setting container_image for all osd 2024-04-02T04:54:26.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:26.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-02T04:54:26.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-02T04:54:26.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-02T04:54:26.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-02T04:54:26.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-02T04:54:26.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-02T04:54:26.148 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-02T04:54:26.148 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-02T04:54:26.148 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-02T04:54:26.148 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-02T04:54:26.148 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-02T04:54:26.148 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-02T04:54:26.148 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-04-02T04:54:26.148 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-04-02T04:54:26.148 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-04-02T04:54:26.148 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-04-02T04:54:26.148 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:25 smithi122 ceph-mon[53211]: Upgrade: Setting require_osd_release to 19 squid 2024-04-02T04:54:26.148 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:25 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-02T04:54:26.340 INFO:teuthology.orchestra.run.smithi053.stdout: 5 275679 9.56 MB/sec execute 576 sec latency 50.301 ms 2024-04-02T04:54:26.644 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:26 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:54:26.223+0000 7f892ccdd7c0 -1 osd.7 138 log_to_monitors true 2024-04-02T04:54:26.907 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:26 smithi053 ceph-mon[67180]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-02T04:54:26.908 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:26 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-02T04:54:26.908 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:26 smithi053 ceph-mon[67180]: osdmap e141: 8 total, 7 up, 8 in 2024-04-02T04:54:26.908 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:26 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi053.nfrfot"]}]: dispatch 2024-04-02T04:54:26.908 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:26 smithi053 ceph-mon[67180]: Upgrade: It appears safe to stop mds.foofs.smithi053.nfrfot 2024-04-02T04:54:26.908 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:26 smithi053 ceph-mon[67180]: from='osd.7 [v2:172.21.15.122:6824/3766693071,v1:172.21.15.122:6825/3766693071]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-02T04:54:26.908 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:26 smithi053 ceph-mon[67180]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-02T04:54:26.908 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:26 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:26.908 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:26 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:54:27.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:26 smithi122 ceph-mon[53211]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-02T04:54:27.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:26 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-02T04:54:27.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:26 smithi122 ceph-mon[53211]: osdmap e141: 8 total, 7 up, 8 in 2024-04-02T04:54:27.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:26 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi053.nfrfot"]}]: dispatch 2024-04-02T04:54:27.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:26 smithi122 ceph-mon[53211]: Upgrade: It appears safe to stop mds.foofs.smithi053.nfrfot 2024-04-02T04:54:27.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:26 smithi122 ceph-mon[53211]: from='osd.7 [v2:172.21.15.122:6824/3766693071,v1:172.21.15.122:6825/3766693071]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-02T04:54:27.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:26 smithi122 ceph-mon[53211]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-02T04:54:27.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:26 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:27.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:26 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:54:27.341 INFO:teuthology.orchestra.run.smithi053.stdout: 5 275905 9.55 MB/sec execute 577 sec latency 87.070 ms 2024-04-02T04:54:27.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:27 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:54:27.596+0000 7f891a12d700 -1 osd.7 138 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T04:54:28.186 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:28 smithi053 ceph-mon[67180]: pgmap v280: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 336 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.2 MiB/s wr, 497 op/s; 239/1566 objects degraded (15.262%) 2024-04-02T04:54:28.186 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:28 smithi053 ceph-mon[67180]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-02T04:54:28.187 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:28 smithi053 ceph-mon[67180]: from='osd.7 [v2:172.21.15.122:6824/3766693071,v1:172.21.15.122:6825/3766693071]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-02T04:54:28.187 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:28 smithi053 ceph-mon[67180]: osdmap e142: 8 total, 7 up, 8 in 2024-04-02T04:54:28.187 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:28 smithi053 ceph-mon[67180]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-02T04:54:28.187 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:28 smithi053 ceph-mon[67180]: from='osd.7 ' entity='osd.7' 2024-04-02T04:54:28.341 INFO:teuthology.orchestra.run.smithi053.stdout: 5 276190 9.55 MB/sec execute 578 sec latency 133.373 ms 2024-04-02T04:54:28.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:28 smithi122 ceph-mon[53211]: pgmap v280: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 336 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.2 MiB/s wr, 497 op/s; 239/1566 objects degraded (15.262%) 2024-04-02T04:54:28.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:28 smithi122 ceph-mon[53211]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-02T04:54:28.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:28 smithi122 ceph-mon[53211]: from='osd.7 [v2:172.21.15.122:6824/3766693071,v1:172.21.15.122:6825/3766693071]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-02T04:54:28.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:28 smithi122 ceph-mon[53211]: osdmap e142: 8 total, 7 up, 8 in 2024-04-02T04:54:28.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:28 smithi122 ceph-mon[53211]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-02T04:54:28.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:28 smithi122 ceph-mon[53211]: from='osd.7 ' entity='osd.7' 2024-04-02T04:54:29.341 INFO:teuthology.orchestra.run.smithi053.stdout: 5 276261 9.54 MB/sec execute 579 sec latency 742.665 ms 2024-04-02T04:54:29.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:29 smithi122 ceph-mon[53211]: Upgrade: Updating mds.foofs.smithi053.nfrfot 2024-04-02T04:54:29.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:29 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:29.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:29 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi053.nfrfot", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T04:54:29.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:29 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:54:29.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:29 smithi122 ceph-mon[53211]: Deploying daemon mds.foofs.smithi053.nfrfot on smithi053 2024-04-02T04:54:29.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:29 smithi122 ceph-mon[53211]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-02T04:54:29.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:29 smithi122 ceph-mon[53211]: osd.7 [v2:172.21.15.122:6824/3766693071,v1:172.21.15.122:6825/3766693071] boot 2024-04-02T04:54:29.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:29 smithi122 ceph-mon[53211]: osdmap e143: 8 total, 8 up, 8 in 2024-04-02T04:54:29.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:29 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:54:29.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:29 smithi053 ceph-mon[67180]: Upgrade: Updating mds.foofs.smithi053.nfrfot 2024-04-02T04:54:29.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:29 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:29.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:29 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi053.nfrfot", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T04:54:29.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:29 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:54:29.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:29 smithi053 ceph-mon[67180]: Deploying daemon mds.foofs.smithi053.nfrfot on smithi053 2024-04-02T04:54:29.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:29 smithi053 ceph-mon[67180]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-02T04:54:29.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:29 smithi053 ceph-mon[67180]: osd.7 [v2:172.21.15.122:6824/3766693071,v1:172.21.15.122:6825/3766693071] boot 2024-04-02T04:54:29.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:29 smithi053 ceph-mon[67180]: osdmap e143: 8 total, 8 up, 8 in 2024-04-02T04:54:29.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:29 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T04:54:30.341 INFO:teuthology.orchestra.run.smithi053.stdout: 5 276261 9.52 MB/sec execute 580 sec latency 1742.697 ms 2024-04-02T04:54:30.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:30 smithi122 ceph-mon[53211]: pgmap v283: 97 pgs: 3 peering, 7 active+undersized, 32 active+undersized+degraded, 55 active+clean; 337 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 3.5 MiB/s wr, 468 op/s; 210/1560 objects degraded (13.462%) 2024-04-02T04:54:30.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:30 smithi122 ceph-mon[53211]: Health check update: Degraded data redundancy: 210/1560 objects degraded (13.462%), 32 pgs degraded (PG_DEGRADED) 2024-04-02T04:54:30.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:30 smithi122 ceph-mon[53211]: osdmap e144: 8 total, 8 up, 8 in 2024-04-02T04:54:30.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:30 smithi053 ceph-mon[67180]: pgmap v283: 97 pgs: 3 peering, 7 active+undersized, 32 active+undersized+degraded, 55 active+clean; 337 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 3.5 MiB/s wr, 468 op/s; 210/1560 objects degraded (13.462%) 2024-04-02T04:54:30.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:30 smithi053 ceph-mon[67180]: Health check update: Degraded data redundancy: 210/1560 objects degraded (13.462%), 32 pgs degraded (PG_DEGRADED) 2024-04-02T04:54:30.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:30 smithi053 ceph-mon[67180]: osdmap e144: 8 total, 8 up, 8 in 2024-04-02T04:54:31.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:30 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi053[67176]: 2024-04-02T04:54:30.818+0000 7f598e14e700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T04:54:31.341 INFO:teuthology.orchestra.run.smithi053.stdout: 5 276261 9.50 MB/sec execute 581 sec latency 2742.730 ms 2024-04-02T04:54:31.395 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:30 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:54:30.917+0000 7f89020fd700 -1 bdev(0x55609d412000 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-04-02T04:54:31.395 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:30 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:54:30.917+0000 7f890890a700 -1 bdev(0x55609d412000 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-04-02T04:54:31.395 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:54:30 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:54:30.921+0000 7f8906105700 -1 bdev(0x55609d412000 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 6 2024-04-02T04:54:31.395 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:54:30 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:54:30.919+0000 7f45a8758700 -1 bdev(0x5631a6bf6000 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 5 2024-04-02T04:54:31.395 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:54:30 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:54:30.923+0000 7f45aaf5d700 -1 bdev(0x5631a6bf6000 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 21 2024-04-02T04:54:31.395 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:54:30 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:54:30.924+0000 7f45a5752700 -1 bdev(0x5631a6bf6000 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 25 2024-04-02T04:54:31.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:31 smithi122 ceph-mon[53211]: osdmap e145: 8 total, 8 up, 8 in 2024-04-02T04:54:31.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:31 smithi122 ceph-mon[53211]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-02T04:54:31.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:31 smithi122 ceph-mon[53211]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T04:54:31.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:31 smithi122 ceph-mon[53211]: osdmap e146: 8 total, 8 up, 8 in 2024-04-02T04:54:31.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:31 smithi122 ceph-mon[53211]: Standby daemon mds.foofs.smithi122.mwoyoh assigned to filesystem foofs as rank 0 2024-04-02T04:54:31.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:31 smithi122 ceph-mon[53211]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-02T04:54:31.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:31 smithi122 ceph-mon[53211]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-02T04:54:31.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:31 smithi122 ceph-mon[53211]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-04-02T04:54:31.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:31 smithi122 ceph-mon[53211]: fsmap foofs:1/1 {0=foofs.smithi122.mwoyoh=up:replay} 2024-04-02T04:54:31.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:31 smithi122 ceph-mon[53211]: pgmap v287: 97 pgs: 3 remapped+peering, 9 peering, 2 active+undersized, 22 active+undersized+degraded, 61 active+clean; 338 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.6 MiB/s wr, 566 op/s; 122/1578 objects degraded (7.731%); 0 B/s, 0 objects/s recovering 2024-04-02T04:54:32.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:31 smithi053 ceph-mon[67180]: osdmap e145: 8 total, 8 up, 8 in 2024-04-02T04:54:32.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:31 smithi053 ceph-mon[67180]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-02T04:54:32.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:31 smithi053 ceph-mon[67180]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T04:54:32.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:31 smithi053 ceph-mon[67180]: osdmap e146: 8 total, 8 up, 8 in 2024-04-02T04:54:32.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:31 smithi053 ceph-mon[67180]: Standby daemon mds.foofs.smithi122.mwoyoh assigned to filesystem foofs as rank 0 2024-04-02T04:54:32.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:31 smithi053 ceph-mon[67180]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-02T04:54:32.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:31 smithi053 ceph-mon[67180]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-02T04:54:32.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:31 smithi053 ceph-mon[67180]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-04-02T04:54:32.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:31 smithi053 ceph-mon[67180]: fsmap foofs:1/1 {0=foofs.smithi122.mwoyoh=up:replay} 2024-04-02T04:54:32.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:31 smithi053 ceph-mon[67180]: pgmap v287: 97 pgs: 3 remapped+peering, 9 peering, 2 active+undersized, 22 active+undersized+degraded, 61 active+clean; 338 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.6 MiB/s wr, 566 op/s; 122/1578 objects degraded (7.731%); 0 B/s, 0 objects/s recovering 2024-04-02T04:54:32.341 INFO:teuthology.orchestra.run.smithi053.stdout: 5 276261 9.49 MB/sec execute 582 sec latency 3742.770 ms 2024-04-02T04:54:33.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:32 smithi122 ceph-mon[53211]: osdmap e147: 8 total, 8 up, 8 in 2024-04-02T04:54:33.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:32 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi053.nfrfot"}]: dispatch 2024-04-02T04:54:33.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:32 smithi122 ceph-mon[53211]: mds.? [v2:172.21.15.122:6832/233414887,v1:172.21.15.122:6833/233414887] up:reconnect 2024-04-02T04:54:33.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:32 smithi122 ceph-mon[53211]: fsmap foofs:1/1 {0=foofs.smithi122.mwoyoh=up:reconnect} 2024-04-02T04:54:33.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:32 smithi053 ceph-mon[67180]: osdmap e147: 8 total, 8 up, 8 in 2024-04-02T04:54:33.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:32 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi053.nfrfot"}]: dispatch 2024-04-02T04:54:33.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:32 smithi053 ceph-mon[67180]: mds.? [v2:172.21.15.122:6832/233414887,v1:172.21.15.122:6833/233414887] up:reconnect 2024-04-02T04:54:33.308 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:32 smithi053 ceph-mon[67180]: fsmap foofs:1/1 {0=foofs.smithi122.mwoyoh=up:reconnect} 2024-04-02T04:54:33.341 INFO:teuthology.orchestra.run.smithi053.stdout: 5 276261 9.47 MB/sec execute 583 sec latency 4742.817 ms 2024-04-02T04:54:34.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:33 smithi122 ceph-mon[53211]: reconnect by client.24369 172.21.15.122:0/514971600 after 0.000999977 2024-04-02T04:54:34.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:33 smithi122 ceph-mon[53211]: reconnect by client.14538 172.21.15.53:0/2653381202 after 0.0609986 2024-04-02T04:54:34.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:33 smithi122 ceph-mon[53211]: osdmap e148: 8 total, 8 up, 8 in 2024-04-02T04:54:34.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:33 smithi122 ceph-mon[53211]: pgmap v290: 97 pgs: 1 remapped+peering, 15 peering, 2 active+undersized, 5 active+undersized+degraded, 74 active+clean; 339 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 34 MiB/s rd, 1.2 MiB/s wr, 91 op/s; 32/1569 objects degraded (2.040%); 0 B/s, 0 objects/s recovering 2024-04-02T04:54:34.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:33 smithi122 ceph-mon[53211]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-02T04:54:34.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:33 smithi122 ceph-mon[53211]: mds.? [v2:172.21.15.122:6832/233414887,v1:172.21.15.122:6833/233414887] up:rejoin 2024-04-02T04:54:34.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:33 smithi122 ceph-mon[53211]: mds.? [v2:172.21.15.53:6834/2288395252,v1:172.21.15.53:6835/2288395252] up:boot 2024-04-02T04:54:34.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:33 smithi122 ceph-mon[53211]: fsmap foofs:1/1 {0=foofs.smithi122.mwoyoh=up:rejoin} 1 up:standby 2024-04-02T04:54:34.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:33 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi053.nfrfot"}]: dispatch 2024-04-02T04:54:34.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:33 smithi053 ceph-mon[67180]: reconnect by client.24369 172.21.15.122:0/514971600 after 0.000999977 2024-04-02T04:54:34.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:33 smithi053 ceph-mon[67180]: reconnect by client.14538 172.21.15.53:0/2653381202 after 0.0609986 2024-04-02T04:54:34.308 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:33 smithi053 ceph-mon[67180]: osdmap e148: 8 total, 8 up, 8 in 2024-04-02T04:54:34.308 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:33 smithi053 ceph-mon[67180]: pgmap v290: 97 pgs: 1 remapped+peering, 15 peering, 2 active+undersized, 5 active+undersized+degraded, 74 active+clean; 339 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 34 MiB/s rd, 1.2 MiB/s wr, 91 op/s; 32/1569 objects degraded (2.040%); 0 B/s, 0 objects/s recovering 2024-04-02T04:54:34.308 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:33 smithi053 ceph-mon[67180]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-02T04:54:34.308 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:33 smithi053 ceph-mon[67180]: mds.? [v2:172.21.15.122:6832/233414887,v1:172.21.15.122:6833/233414887] up:rejoin 2024-04-02T04:54:34.308 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:33 smithi053 ceph-mon[67180]: mds.? [v2:172.21.15.53:6834/2288395252,v1:172.21.15.53:6835/2288395252] up:boot 2024-04-02T04:54:34.308 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:33 smithi053 ceph-mon[67180]: fsmap foofs:1/1 {0=foofs.smithi122.mwoyoh=up:rejoin} 1 up:standby 2024-04-02T04:54:34.308 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:33 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi053.nfrfot"}]: dispatch 2024-04-02T04:54:34.341 INFO:teuthology.orchestra.run.smithi053.stdout: 5 276261 9.45 MB/sec execute 584 sec latency 5742.862 ms 2024-04-02T04:54:35.341 INFO:teuthology.orchestra.run.smithi053.stdout: 5 276262 9.44 MB/sec execute 585 sec latency 6742.902 ms 2024-04-02T04:54:35.889 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:35 smithi053 ceph-mon[67180]: mds.? [v2:172.21.15.122:6832/233414887,v1:172.21.15.122:6833/233414887] up:clientreplay 2024-04-02T04:54:35.889 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:35 smithi053 ceph-mon[67180]: fsmap foofs:1/1 {0=foofs.smithi122.mwoyoh=up:clientreplay} 1 up:standby 2024-04-02T04:54:35.889 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:35 smithi053 ceph-mon[67180]: daemon mds.foofs.smithi122.mwoyoh is now active in filesystem foofs as rank 0 2024-04-02T04:54:35.889 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:35 smithi053 ceph-mon[67180]: pgmap v291: 97 pgs: 1 remapped+peering, 6 peering, 90 active+clean; 339 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 37 MiB/s rd, 742 KiB/s wr, 60 op/s; 2.9 MiB/s, 27 objects/s recovering 2024-04-02T04:54:35.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:35 smithi122 ceph-mon[53211]: mds.? [v2:172.21.15.122:6832/233414887,v1:172.21.15.122:6833/233414887] up:clientreplay 2024-04-02T04:54:35.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:35 smithi122 ceph-mon[53211]: fsmap foofs:1/1 {0=foofs.smithi122.mwoyoh=up:clientreplay} 1 up:standby 2024-04-02T04:54:35.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:35 smithi122 ceph-mon[53211]: daemon mds.foofs.smithi122.mwoyoh is now active in filesystem foofs as rank 0 2024-04-02T04:54:35.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:35 smithi122 ceph-mon[53211]: pgmap v291: 97 pgs: 1 remapped+peering, 6 peering, 90 active+clean; 339 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 37 MiB/s rd, 742 KiB/s wr, 60 op/s; 2.9 MiB/s, 27 objects/s recovering 2024-04-02T04:54:36.341 INFO:teuthology.orchestra.run.smithi053.stdout: 5 276406 9.43 MB/sec execute 586 sec latency 7080.145 ms 2024-04-02T04:54:36.660 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:36 smithi053 ceph-mon[67180]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 32/1569 objects degraded (2.040%), 5 pgs degraded) 2024-04-02T04:54:36.660 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:36 smithi053 ceph-mon[67180]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-02T04:54:36.660 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:36 smithi053 ceph-mon[67180]: mds.? [v2:172.21.15.122:6832/233414887,v1:172.21.15.122:6833/233414887] up:active 2024-04-02T04:54:36.660 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:36 smithi053 ceph-mon[67180]: fsmap foofs:1 {0=foofs.smithi122.mwoyoh=up:active} 1 up:standby 2024-04-02T04:54:36.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:36 smithi122 ceph-mon[53211]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 32/1569 objects degraded (2.040%), 5 pgs degraded) 2024-04-02T04:54:36.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:36 smithi122 ceph-mon[53211]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-02T04:54:36.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:36 smithi122 ceph-mon[53211]: mds.? [v2:172.21.15.122:6832/233414887,v1:172.21.15.122:6833/233414887] up:active 2024-04-02T04:54:36.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:36 smithi122 ceph-mon[53211]: fsmap foofs:1 {0=foofs.smithi122.mwoyoh=up:active} 1 up:standby 2024-04-02T04:54:37.341 INFO:teuthology.orchestra.run.smithi053.stdout: 5 276554 9.43 MB/sec execute 587 sec latency 62.527 ms 2024-04-02T04:54:37.675 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:37 smithi053 ceph-mon[67180]: pgmap v292: 97 pgs: 6 peering, 91 active+clean; 338 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 38 MiB/s rd, 526 KiB/s wr, 55 op/s; 2.1 MiB/s, 23 objects/s recovering 2024-04-02T04:54:37.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:37 smithi122 ceph-mon[53211]: pgmap v292: 97 pgs: 6 peering, 91 active+clean; 338 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 38 MiB/s rd, 526 KiB/s wr, 55 op/s; 2.1 MiB/s, 23 objects/s recovering 2024-04-02T04:54:38.341 INFO:teuthology.orchestra.run.smithi053.stdout: 5 276765 9.42 MB/sec execute 588 sec latency 62.148 ms 2024-04-02T04:54:39.341 INFO:teuthology.orchestra.run.smithi053.stdout: 5 276949 9.42 MB/sec execute 589 sec latency 107.055 ms 2024-04-02T04:54:40.018 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:39 smithi053 ceph-mon[67180]: pgmap v293: 97 pgs: 97 active+clean; 341 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 29 MiB/s rd, 1.1 MiB/s wr, 112 op/s; 2.1 MiB/s, 36 objects/s recovering 2024-04-02T04:54:40.341 INFO:teuthology.orchestra.run.smithi053.stdout: 5 277145 9.42 MB/sec execute 590 sec latency 82.758 ms 2024-04-02T04:54:40.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:39 smithi122 ceph-mon[53211]: pgmap v293: 97 pgs: 97 active+clean; 341 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 29 MiB/s rd, 1.1 MiB/s wr, 112 op/s; 2.1 MiB/s, 36 objects/s recovering 2024-04-02T04:54:41.341 INFO:teuthology.orchestra.run.smithi053.stdout: 5 277312 9.41 MB/sec execute 591 sec latency 83.605 ms 2024-04-02T04:54:42.064 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:41 smithi053 ceph-mon[67180]: pgmap v294: 97 pgs: 97 active+clean; 343 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 1.4 MiB/s wr, 109 op/s; 1.9 MiB/s, 32 objects/s recovering 2024-04-02T04:54:42.064 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:41 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:42.065 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:41 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:54:42.341 INFO:teuthology.orchestra.run.smithi053.stdout: 5 277437 9.40 MB/sec execute 592 sec latency 122.447 ms 2024-04-02T04:54:42.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:41 smithi122 ceph-mon[53211]: pgmap v294: 97 pgs: 97 active+clean; 343 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 1.4 MiB/s wr, 109 op/s; 1.9 MiB/s, 32 objects/s recovering 2024-04-02T04:54:42.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:41 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:42.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:41 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:54:43.341 INFO:teuthology.orchestra.run.smithi053.stdout: 5 277628 9.38 MB/sec execute 593 sec latency 60.279 ms 2024-04-02T04:54:44.102 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:43 smithi053 ceph-mon[67180]: pgmap v295: 97 pgs: 97 active+clean; 345 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.7 MiB/s wr, 159 op/s; 1.7 MiB/s, 29 objects/s recovering 2024-04-02T04:54:44.103 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:43 smithi053 ceph-mon[67180]: daemon mds.foofs.smithi053.nfrfot restarted 2024-04-02T04:54:44.103 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:43 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:44.103 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:43 smithi053 ceph-mon[67180]: osdmap e149: 8 total, 8 up, 8 in 2024-04-02T04:54:44.103 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:43 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:44.103 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:43 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:54:44.342 INFO:teuthology.orchestra.run.smithi053.stdout: 5 277944 9.38 MB/sec execute 594 sec latency 58.463 ms 2024-04-02T04:54:44.394 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:43 smithi122 ceph-mon[53211]: pgmap v295: 97 pgs: 97 active+clean; 345 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.7 MiB/s wr, 159 op/s; 1.7 MiB/s, 29 objects/s recovering 2024-04-02T04:54:44.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:43 smithi122 ceph-mon[53211]: daemon mds.foofs.smithi053.nfrfot restarted 2024-04-02T04:54:44.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:43 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:44.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:43 smithi122 ceph-mon[53211]: osdmap e149: 8 total, 8 up, 8 in 2024-04-02T04:54:44.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:43 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:44.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:43 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:54:45.049 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:44 smithi053 ceph-mon[67180]: mds.? [v2:172.21.15.53:6834/2789998387,v1:172.21.15.53:6835/2789998387] up:boot 2024-04-02T04:54:45.050 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:44 smithi053 ceph-mon[67180]: fsmap foofs:1 {0=foofs.smithi122.mwoyoh=up:active} 1 up:standby 2024-04-02T04:54:45.050 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:44 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi053.nfrfot"}]: dispatch 2024-04-02T04:54:45.341 INFO:teuthology.orchestra.run.smithi053.stdout: 5 278267 9.38 MB/sec execute 595 sec latency 57.504 ms 2024-04-02T04:54:45.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:44 smithi122 ceph-mon[53211]: mds.? [v2:172.21.15.53:6834/2789998387,v1:172.21.15.53:6835/2789998387] up:boot 2024-04-02T04:54:45.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:44 smithi122 ceph-mon[53211]: fsmap foofs:1 {0=foofs.smithi122.mwoyoh=up:active} 1 up:standby 2024-04-02T04:54:45.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:44 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi053.nfrfot"}]: dispatch 2024-04-02T04:54:46.173 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:45 smithi053 ceph-mon[67180]: pgmap v297: 97 pgs: 97 active+clean; 348 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.2 MiB/s rd, 2.1 MiB/s wr, 197 op/s; 469 KiB/s, 17 objects/s recovering 2024-04-02T04:54:46.296 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:45 smithi122 ceph-mon[53211]: pgmap v297: 97 pgs: 97 active+clean; 348 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.2 MiB/s rd, 2.1 MiB/s wr, 197 op/s; 469 KiB/s, 17 objects/s recovering 2024-04-02T04:54:46.341 INFO:teuthology.orchestra.run.smithi053.stdout: 5 278618 9.38 MB/sec execute 596 sec latency 56.418 ms 2024-04-02T04:54:47.341 INFO:teuthology.orchestra.run.smithi053.stdout: 5 279000 9.38 MB/sec execute 597 sec latency 63.906 ms 2024-04-02T04:54:47.828 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:47 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:47.829 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:47 smithi053 ceph-mon[67180]: pgmap v298: 97 pgs: 97 active+clean; 343 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 33 KiB/s rd, 2.2 MiB/s wr, 205 op/s; 442 KiB/s, 14 objects/s recovering 2024-04-02T04:54:47.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:47 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:47.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:47 smithi122 ceph-mon[53211]: pgmap v298: 97 pgs: 97 active+clean; 343 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 33 KiB/s rd, 2.2 MiB/s wr, 205 op/s; 442 KiB/s, 14 objects/s recovering 2024-04-02T04:54:48.341 INFO:teuthology.orchestra.run.smithi053.stdout: 5 279471 9.37 MB/sec execute 598 sec latency 52.701 ms 2024-04-02T04:54:49.341 INFO:teuthology.orchestra.run.smithi053.stdout: 5 279858 9.38 MB/sec execute 599 sec latency 32.970 ms 2024-04-02T04:54:49.977 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:49 smithi053 ceph-mon[67180]: pgmap v299: 97 pgs: 97 active+clean; 343 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 33 KiB/s rd, 3.1 MiB/s wr, 303 op/s 2024-04-02T04:54:50.341 INFO:teuthology.orchestra.run.smithi053.stdout: 5 cleanup 600 sec 2024-04-02T04:54:50.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:49 smithi122 ceph-mon[53211]: pgmap v299: 97 pgs: 97 active+clean; 343 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 33 KiB/s rd, 3.1 MiB/s wr, 303 op/s 2024-04-02T04:54:50.650 INFO:teuthology.orchestra.run.smithi053.stdout: 0 cleanup 600 sec 2024-04-02T04:54:50.651 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-02T04:54:50.651 INFO:teuthology.orchestra.run.smithi053.stdout: Operation Count AvgLat MaxLat 2024-04-02T04:54:50.651 INFO:teuthology.orchestra.run.smithi053.stdout: ---------------------------------------- 2024-04-02T04:54:50.651 INFO:teuthology.orchestra.run.smithi053.stdout: NTCreateX 181032 4.776 7041.331 2024-04-02T04:54:50.651 INFO:teuthology.orchestra.run.smithi053.stdout: Close 132939 0.140 12.679 2024-04-02T04:54:50.651 INFO:teuthology.orchestra.run.smithi053.stdout: Rename 7674 9.921 133.363 2024-04-02T04:54:50.651 INFO:teuthology.orchestra.run.smithi053.stdout: Unlink 36603 6.935 1038.750 2024-04-02T04:54:50.651 INFO:teuthology.orchestra.run.smithi053.stdout: Qpathinfo 164193 1.219 1039.097 2024-04-02T04:54:50.651 INFO:teuthology.orchestra.run.smithi053.stdout: Qfileinfo 28623 0.008 0.754 2024-04-02T04:54:50.651 INFO:teuthology.orchestra.run.smithi053.stdout: Qfsinfo 30088 0.653 20.537 2024-04-02T04:54:50.652 INFO:teuthology.orchestra.run.smithi053.stdout: Sfileinfo 14784 12.981 1039.773 2024-04-02T04:54:50.652 INFO:teuthology.orchestra.run.smithi053.stdout: Find 63429 1.008 21.279 2024-04-02T04:54:50.652 INFO:teuthology.orchestra.run.smithi053.stdout: WriteX 89421 14.586 7080.142 2024-04-02T04:54:50.652 INFO:teuthology.orchestra.run.smithi053.stdout: ReadX 283977 0.010 3.462 2024-04-02T04:54:50.652 INFO:teuthology.orchestra.run.smithi053.stdout: LockX 590 0.410 8.136 2024-04-02T04:54:50.652 INFO:teuthology.orchestra.run.smithi053.stdout: UnlockX 590 0.398 12.543 2024-04-02T04:54:50.652 INFO:teuthology.orchestra.run.smithi053.stdout: Flush 12704 0.002 0.033 2024-04-02T04:54:50.652 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-02T04:54:50.652 INFO:teuthology.orchestra.run.smithi053.stdout:Throughput 9.37839 MB/sec 5 clients 5 procs max_latency=7080.145 ms 2024-04-02T04:54:50.653 DEBUG:teuthology.orchestra.run.smithi053:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'umount /mnt/foo' 2024-04-02T04:54:50.694 DEBUG:teuthology.orchestra.run.smithi053:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done' 2024-04-02T04:54:50.812 DEBUG:teuthology.orchestra.run.smithi053:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 5' 2024-04-02T04:54:50.839 INFO:teuthology.orchestra.run.smithi053.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-02T04:54:50.839 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-02T04:54:50.839 INFO:teuthology.orchestra.run.smithi053.stdout:Running for 5 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 1 secs 2024-04-02T04:54:50.840 INFO:teuthology.orchestra.run.smithi053.stdout:3 of 5 processes prepared for launch 0 sec 2024-04-02T04:54:50.940 INFO:teuthology.orchestra.run.smithi053.stdout:5 of 5 processes prepared for launch 0 sec 2024-04-02T04:54:50.940 INFO:teuthology.orchestra.run.smithi053.stdout:releasing clients 2024-04-02T04:54:52.163 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:51 smithi053 ceph-mon[67180]: pgmap v300: 97 pgs: 97 active+clean; 345 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 3.7 MiB/s wr, 429 op/s 2024-04-02T04:54:52.163 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:51 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:52.163 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:51 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:52.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:51 smithi122 ceph-mon[53211]: pgmap v300: 97 pgs: 97 active+clean; 345 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 3.7 MiB/s wr, 429 op/s 2024-04-02T04:54:52.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:51 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:52.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:51 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:52.940 INFO:teuthology.orchestra.run.smithi053.stdout: 5 332 18.21 MB/sec execute 1 sec latency 89.501 ms 2024-04-02T04:54:53.940 INFO:teuthology.orchestra.run.smithi053.stdout: 5 435 17.43 MB/sec execute 2 sec latency 76.738 ms 2024-04-02T04:54:54.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:54 smithi053 ceph-mon[67180]: pgmap v301: 97 pgs: 97 active+clean; 331 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 6.1 MiB/s wr, 474 op/s 2024-04-02T04:54:54.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:54.308 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:54.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:54 smithi122 ceph-mon[53211]: pgmap v301: 97 pgs: 97 active+clean; 331 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 6.1 MiB/s wr, 474 op/s 2024-04-02T04:54:54.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:54.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:54.706 INFO:teuthology.orchestra.run.smithi053.stdout:true 2024-04-02T04:54:54.940 INFO:teuthology.orchestra.run.smithi053.stdout: 5 541 17.90 MB/sec execute 3 sec latency 86.997 ms 2024-04-02T04:54:55.618 INFO:teuthology.orchestra.run.smithi053.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-02T04:54:55.618 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager.smithi053 smithi053 *:9093,9094 running (9m) 3s ago 18m 23.8M - 0.23.0 ba2b418f427c bf50c1cd2b3d 2024-04-02T04:54:55.618 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi053 smithi053 running (5m) 3s ago 18m 6660k - 19.0.0-1979-g6cec793d 78f830190839 4ce8915e0157 2024-04-02T04:54:55.618 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi122 smithi122 running (5m) 36s ago 17m 6660k - 19.0.0-1979-g6cec793d 78f830190839 e2b5369e6fdf 2024-04-02T04:54:55.618 INFO:teuthology.orchestra.run.smithi053.stdout:grafana.smithi053 smithi053 *:3000 error 3s ago 18m - - 2024-04-02T04:54:55.618 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi053.nfrfot smithi053 running (12s) 3s ago 12m 14.8M - 19.0.0-1979-g6cec793d 78f830190839 e7c0900d609e 2024-04-02T04:54:55.618 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi122.mwoyoh smithi122 running (12m) 36s ago 12m 16.5M - 17.2.0 e1d6a67b021e 73376d9ee4b8 2024-04-02T04:54:55.618 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi053.legdmv smithi053 *:8443,9283,8765 running (8m) 3s ago 19m 511M - 19.0.0-1979-g6cec793d 78f830190839 c12d516a31bc 2024-04-02T04:54:55.618 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi122.cpwlyk smithi122 *:8443,9283,8765 running (7m) 36s ago 17m 452M - 19.0.0-1979-g6cec793d 78f830190839 9a545c5e7d81 2024-04-02T04:54:55.619 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi053 smithi053 running (7m) 3s ago 19m 59.5M 2048M 19.0.0-1979-g6cec793d 78f830190839 f53a44f19c00 2024-04-02T04:54:55.619 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi122 smithi122 running (6m) 36s ago 17m 42.8M 2048M 19.0.0-1979-g6cec793d 78f830190839 44b200fed3c5 2024-04-02T04:54:55.619 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.0.0.smithi053.mhkdcb smithi053 *:2049 running (12m) 3s ago 12m 301M - 4.0 e1d6a67b021e 07e5da7bd2c5 2024-04-02T04:54:55.619 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.1.0.smithi122.mauvsb smithi122 *:2049 running (12m) 36s ago 12m 84.3M - 4.0 e1d6a67b021e 4f866685005e 2024-04-02T04:54:55.619 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi053 smithi053 *:9100 running (9m) 3s ago 18m 18.1M - 1.3.1 1dbe0e931976 773b71971e67 2024-04-02T04:54:55.619 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi122 smithi122 *:9100 running (8m) 36s ago 17m 16.3M - 1.3.1 1dbe0e931976 475d439c7980 2024-04-02T04:54:55.619 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 smithi053 running (5m) 3s ago 17m 391M 1937M 19.0.0-1979-g6cec793d 78f830190839 490c346c7639 2024-04-02T04:54:55.619 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 smithi053 running (4m) 3s ago 16m 377M 1937M 19.0.0-1979-g6cec793d 78f830190839 e70e40949c2f 2024-04-02T04:54:55.619 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 smithi053 running (3m) 3s ago 16m 365M 1937M 19.0.0-1979-g6cec793d 78f830190839 4f053827e0dd 2024-04-02T04:54:55.619 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 smithi053 running (3m) 3s ago 15m 411M 1937M 19.0.0-1979-g6cec793d 78f830190839 5cd6de7a6535 2024-04-02T04:54:55.619 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 smithi122 running (2m) 36s ago 15m 328M 2704M 19.0.0-1979-g6cec793d 78f830190839 40d6ab1aac4b 2024-04-02T04:54:55.619 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 smithi122 running (115s) 36s ago 14m 311M 2704M 19.0.0-1979-g6cec793d 78f830190839 be26f425f51b 2024-04-02T04:54:55.619 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 smithi122 running (79s) 36s ago 14m 262M 2704M 19.0.0-1979-g6cec793d 78f830190839 aab18af876da 2024-04-02T04:54:55.619 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 smithi122 running (41s) 36s ago 14m 12.4M 2704M 19.0.0-1979-g6cec793d 78f830190839 86ee236f9426 2024-04-02T04:54:55.619 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus.smithi053 smithi053 *:9095 running (7m) 3s ago 17m 71.9M - 2.33.4 514e6a882f6e 1dd55399da10 2024-04-02T04:54:55.940 INFO:teuthology.orchestra.run.smithi053.stdout: 5 744 16.20 MB/sec execute 4 sec latency 106.699 ms 2024-04-02T04:54:56.205 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:54:56.206 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": { 2024-04-02T04:54:56.206 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:54:56.206 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:54:56.206 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": { 2024-04-02T04:54:56.206 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:54:56.206 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:54:56.206 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": { 2024-04-02T04:54:56.206 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 8 2024-04-02T04:54:56.206 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:54:56.206 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": { 2024-04-02T04:54:56.206 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-02T04:54:56.206 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 1 2024-04-02T04:54:56.207 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:54:56.207 INFO:teuthology.orchestra.run.smithi053.stdout: "overall": { 2024-04-02T04:54:56.207 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-02T04:54:56.207 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 13 2024-04-02T04:54:56.207 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-02T04:54:56.207 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:54:56.493 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:56 smithi053 ceph-mon[67180]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:54:56.493 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:56 smithi053 ceph-mon[67180]: pgmap v302: 97 pgs: 97 active+clean; 310 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 23 KiB/s rd, 8.7 MiB/s wr, 505 op/s 2024-04-02T04:54:56.493 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:56 smithi053 ceph-mon[67180]: from='client.34444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:54:56.493 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:56 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/1630309284' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:54:56.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:56 smithi122 ceph-mon[53211]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:54:56.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:56 smithi122 ceph-mon[53211]: pgmap v302: 97 pgs: 97 active+clean; 310 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 23 KiB/s rd, 8.7 MiB/s wr, 505 op/s 2024-04-02T04:54:56.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:56 smithi122 ceph-mon[53211]: from='client.34444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:54:56.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:56 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/1630309284' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:54:56.717 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:54:56.717 INFO:teuthology.orchestra.run.smithi053.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae", 2024-04-02T04:54:56.717 INFO:teuthology.orchestra.run.smithi053.stdout: "in_progress": true, 2024-04-02T04:54:56.717 INFO:teuthology.orchestra.run.smithi053.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-02T04:54:56.717 INFO:teuthology.orchestra.run.smithi053.stdout: "services_complete": [ 2024-04-02T04:54:56.717 INFO:teuthology.orchestra.run.smithi053.stdout: "osd", 2024-04-02T04:54:56.717 INFO:teuthology.orchestra.run.smithi053.stdout: "mon", 2024-04-02T04:54:56.717 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr", 2024-04-02T04:54:56.718 INFO:teuthology.orchestra.run.smithi053.stdout: "crash" 2024-04-02T04:54:56.718 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-04-02T04:54:56.718 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": "15/23 daemons upgraded", 2024-04-02T04:54:56.718 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "Currently upgrading mds daemons", 2024-04-02T04:54:56.718 INFO:teuthology.orchestra.run.smithi053.stdout: "is_paused": false 2024-04-02T04:54:56.718 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:54:56.940 INFO:teuthology.orchestra.run.smithi053.stdout: 5 cleanup 5 sec 2024-04-02T04:54:57.295 INFO:teuthology.orchestra.run.smithi053.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-02T04:54:57.295 INFO:teuthology.orchestra.run.smithi053.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-02T04:54:57.295 INFO:teuthology.orchestra.run.smithi053.stdout: daemon grafana.smithi053 on smithi053 is in error state 2024-04-02T04:54:57.497 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:57 smithi053 ceph-mon[67180]: from='client.34448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:54:57.497 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:57 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:54:57.564 INFO:teuthology.orchestra.run.smithi053.stdout: 0 cleanup 6 sec 2024-04-02T04:54:57.564 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-02T04:54:57.564 INFO:teuthology.orchestra.run.smithi053.stdout: Operation Count AvgLat MaxLat 2024-04-02T04:54:57.564 INFO:teuthology.orchestra.run.smithi053.stdout: ---------------------------------------- 2024-04-02T04:54:57.564 INFO:teuthology.orchestra.run.smithi053.stdout: NTCreateX 503 18.199 106.695 2024-04-02T04:54:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: Close 470 11.203 105.761 2024-04-02T04:54:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: Rename 16 35.999 61.427 2024-04-02T04:54:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: Unlink 39 5.634 30.635 2024-04-02T04:54:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: Qpathinfo 115 2.462 34.267 2024-04-02T04:54:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: Qfileinfo 301 20.601 103.685 2024-04-02T04:54:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: Qfsinfo 79 0.639 6.251 2024-04-02T04:54:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: Sfileinfo 76 18.718 33.292 2024-04-02T04:54:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: Find 100 1.091 4.588 2024-04-02T04:54:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: WriteX 1442 0.028 0.187 2024-04-02T04:54:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: ReadX 370 0.032 2.737 2024-04-02T04:54:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: Flush 109 16.578 31.978 2024-04-02T04:54:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-02T04:54:57.565 INFO:teuthology.orchestra.run.smithi053.stdout:Throughput 16.2012 MB/sec 5 clients 5 procs max_latency=106.699 ms 2024-04-02T04:54:57.566 DEBUG:teuthology.parallel:result is None 2024-04-02T04:54:57.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:57 smithi122 ceph-mon[53211]: from='client.34448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:54:57.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:57 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:54:58.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:58 smithi122 ceph-mon[53211]: from='client.44337 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:54:58.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:58 smithi122 ceph-mon[53211]: pgmap v303: 97 pgs: 97 active+clean; 302 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 9.7 MiB/s wr, 517 op/s 2024-04-02T04:54:58.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:58 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/2030468797' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:54:58.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:58 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:58.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:58 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:58.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:58 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:54:58.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:58 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:54:58.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:58 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:58.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:58 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:58.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:58 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:54:58.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:58 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:54:58.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:58 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:54:58.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:58 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:54:58.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:58 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:54:58.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:58 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi122.mwoyoh"]}]: dispatch 2024-04-02T04:54:58.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:58 smithi053 ceph-mon[67180]: from='client.44337 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:54:58.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:58 smithi053 ceph-mon[67180]: pgmap v303: 97 pgs: 97 active+clean; 302 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 9.7 MiB/s wr, 517 op/s 2024-04-02T04:54:58.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:58 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/2030468797' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:54:58.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:58 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:58.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:58 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:58.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:58 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:54:58.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:58 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:54:58.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:58 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:58.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:58 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:54:58.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:58 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:54:58.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:58 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:54:58.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:58 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:54:58.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:58 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:54:58.559 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:58 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:54:58.559 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:58 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi122.mwoyoh"]}]: dispatch 2024-04-02T04:54:59.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:59 smithi122 ceph-mon[53211]: Detected new or changed devices on smithi053 2024-04-02T04:54:59.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:59 smithi122 ceph-mon[53211]: pgmap v304: 97 pgs: 97 active+clean; 312 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 12 MiB/s wr, 576 op/s 2024-04-02T04:54:59.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:59 smithi122 ceph-mon[53211]: pgmap v305: 97 pgs: 97 active+clean; 312 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 13 MiB/s wr, 534 op/s 2024-04-02T04:54:59.396 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:54:59 smithi122 ceph-mon[53211]: Upgrade: It appears safe to stop mds.foofs.smithi122.mwoyoh 2024-04-02T04:54:59.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:59 smithi053 ceph-mon[67180]: Detected new or changed devices on smithi053 2024-04-02T04:54:59.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:59 smithi053 ceph-mon[67180]: pgmap v304: 97 pgs: 97 active+clean; 312 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 12 MiB/s wr, 576 op/s 2024-04-02T04:54:59.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:59 smithi053 ceph-mon[67180]: pgmap v305: 97 pgs: 97 active+clean; 312 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 13 MiB/s wr, 534 op/s 2024-04-02T04:54:59.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:54:59 smithi053 ceph-mon[67180]: Upgrade: It appears safe to stop mds.foofs.smithi122.mwoyoh 2024-04-02T04:55:00.878 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:00 smithi122 ceph-mon[53211]: Upgrade: Updating mds.foofs.smithi122.mwoyoh 2024-04-02T04:55:00.878 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:00 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:00.879 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:00 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi122.mwoyoh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T04:55:00.879 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:00 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:55:00.879 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:00 smithi122 ceph-mon[53211]: Deploying daemon mds.foofs.smithi122.mwoyoh on smithi122 2024-04-02T04:55:00.879 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:00 smithi122 ceph-mon[53211]: pgmap v306: 97 pgs: 97 active+clean; 310 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 14 MiB/s wr, 408 op/s 2024-04-02T04:55:01.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:00 smithi053 ceph-mon[67180]: Upgrade: Updating mds.foofs.smithi122.mwoyoh 2024-04-02T04:55:01.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:00 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:01.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:00 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi122.mwoyoh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T04:55:01.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:00 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:55:01.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:00 smithi053 ceph-mon[67180]: Deploying daemon mds.foofs.smithi122.mwoyoh on smithi122 2024-04-02T04:55:01.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:00 smithi053 ceph-mon[67180]: pgmap v306: 97 pgs: 97 active+clean; 310 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 14 MiB/s wr, 408 op/s 2024-04-02T04:55:02.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:01 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi053[67176]: 2024-04-02T04:55:01.761+0000 7f598e14e700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T04:55:02.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:01 smithi053 ceph-mon[67180]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-02T04:55:02.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:01 smithi053 ceph-mon[67180]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T04:55:02.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:01 smithi053 ceph-mon[67180]: osdmap e150: 8 total, 8 up, 8 in 2024-04-02T04:55:02.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:01 smithi053 ceph-mon[67180]: Standby daemon mds.foofs.smithi053.nfrfot assigned to filesystem foofs as rank 0 2024-04-02T04:55:02.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:01 smithi053 ceph-mon[67180]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-02T04:55:02.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:01 smithi053 ceph-mon[67180]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-02T04:55:02.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:01 smithi053 ceph-mon[67180]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-04-02T04:55:02.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:01 smithi053 ceph-mon[67180]: fsmap foofs:1/1 {0=foofs.smithi053.nfrfot=up:replay} 2024-04-02T04:55:02.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:01 smithi122 ceph-mon[53211]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-02T04:55:02.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:01 smithi122 ceph-mon[53211]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T04:55:02.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:01 smithi122 ceph-mon[53211]: osdmap e150: 8 total, 8 up, 8 in 2024-04-02T04:55:02.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:01 smithi122 ceph-mon[53211]: Standby daemon mds.foofs.smithi053.nfrfot assigned to filesystem foofs as rank 0 2024-04-02T04:55:02.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:01 smithi122 ceph-mon[53211]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-02T04:55:02.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:01 smithi122 ceph-mon[53211]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-02T04:55:02.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:01 smithi122 ceph-mon[53211]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-04-02T04:55:02.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:01 smithi122 ceph-mon[53211]: fsmap foofs:1/1 {0=foofs.smithi053.nfrfot=up:replay} 2024-04-02T04:55:03.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:02 smithi122 ceph-mon[53211]: pgmap v308: 97 pgs: 97 active+clean; 295 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 589 KiB/s rd, 8.1 MiB/s wr, 248 op/s 2024-04-02T04:55:03.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:02 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi122.mwoyoh"}]: dispatch 2024-04-02T04:55:03.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:02 smithi053 ceph-mon[67180]: pgmap v308: 97 pgs: 97 active+clean; 295 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 589 KiB/s rd, 8.1 MiB/s wr, 248 op/s 2024-04-02T04:55:03.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:02 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi122.mwoyoh"}]: dispatch 2024-04-02T04:55:04.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:03 smithi122 ceph-mon[53211]: reconnect by client.14538 172.21.15.53:0/2653381202 after 0 2024-04-02T04:55:04.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:03 smithi122 ceph-mon[53211]: reconnect by client.24369 172.21.15.122:0/514971600 after 0 2024-04-02T04:55:04.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:03 smithi122 ceph-mon[53211]: mds.? [v2:172.21.15.53:6834/2789998387,v1:172.21.15.53:6835/2789998387] up:reconnect 2024-04-02T04:55:04.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:03 smithi122 ceph-mon[53211]: fsmap foofs:1/1 {0=foofs.smithi053.nfrfot=up:reconnect} 2024-04-02T04:55:04.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:03 smithi053 ceph-mon[67180]: reconnect by client.14538 172.21.15.53:0/2653381202 after 0 2024-04-02T04:55:04.308 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:03 smithi053 ceph-mon[67180]: reconnect by client.24369 172.21.15.122:0/514971600 after 0 2024-04-02T04:55:04.308 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:03 smithi053 ceph-mon[67180]: mds.? [v2:172.21.15.53:6834/2789998387,v1:172.21.15.53:6835/2789998387] up:reconnect 2024-04-02T04:55:04.308 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:03 smithi053 ceph-mon[67180]: fsmap foofs:1/1 {0=foofs.smithi053.nfrfot=up:reconnect} 2024-04-02T04:55:05.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:04 smithi122 ceph-mon[53211]: mds.? [v2:172.21.15.53:6834/2789998387,v1:172.21.15.53:6835/2789998387] up:rejoin 2024-04-02T04:55:05.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:04 smithi122 ceph-mon[53211]: fsmap foofs:1/1 {0=foofs.smithi053.nfrfot=up:rejoin} 2024-04-02T04:55:05.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:04 smithi122 ceph-mon[53211]: daemon mds.foofs.smithi053.nfrfot is now active in filesystem foofs as rank 0 2024-04-02T04:55:05.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:04 smithi122 ceph-mon[53211]: pgmap v309: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 5.3 MiB/s wr, 181 op/s 2024-04-02T04:55:05.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:04 smithi053 ceph-mon[67180]: mds.? [v2:172.21.15.53:6834/2789998387,v1:172.21.15.53:6835/2789998387] up:rejoin 2024-04-02T04:55:05.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:04 smithi053 ceph-mon[67180]: fsmap foofs:1/1 {0=foofs.smithi053.nfrfot=up:rejoin} 2024-04-02T04:55:05.308 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:04 smithi053 ceph-mon[67180]: daemon mds.foofs.smithi053.nfrfot is now active in filesystem foofs as rank 0 2024-04-02T04:55:05.308 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:04 smithi053 ceph-mon[67180]: pgmap v309: 97 pgs: 97 active+clean; 231 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 5.3 MiB/s wr, 181 op/s 2024-04-02T04:55:05.983 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:05 smithi122 ceph-mon[53211]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-02T04:55:05.983 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:05 smithi122 ceph-mon[53211]: mds.? [v2:172.21.15.53:6834/2789998387,v1:172.21.15.53:6835/2789998387] up:active 2024-04-02T04:55:05.983 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:05 smithi122 ceph-mon[53211]: fsmap foofs:1 {0=foofs.smithi053.nfrfot=up:active} 2024-04-02T04:55:06.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:05 smithi053 ceph-mon[67180]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-02T04:55:06.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:05 smithi053 ceph-mon[67180]: mds.? [v2:172.21.15.53:6834/2789998387,v1:172.21.15.53:6835/2789998387] up:active 2024-04-02T04:55:06.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:05 smithi053 ceph-mon[67180]: fsmap foofs:1 {0=foofs.smithi053.nfrfot=up:active} 2024-04-02T04:55:06.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:06 smithi122 ceph-mon[53211]: pgmap v310: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 2.6 MiB/s wr, 122 op/s 2024-04-02T04:55:07.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:06 smithi053 ceph-mon[67180]: pgmap v310: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 2.6 MiB/s wr, 122 op/s 2024-04-02T04:55:08.390 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:08 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:08.390 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:08 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:08.390 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:08 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:55:08.390 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:08 smithi122 ceph-mon[53211]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-02T04:55:08.391 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:08 smithi122 ceph-mon[53211]: mds.? [v2:172.21.15.122:6832/481906381,v1:172.21.15.122:6833/481906381] up:boot 2024-04-02T04:55:08.391 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:08 smithi122 ceph-mon[53211]: fsmap foofs:1 {0=foofs.smithi053.nfrfot=up:active} 1 up:standby 2024-04-02T04:55:08.391 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:08 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi122.mwoyoh"}]: dispatch 2024-04-02T04:55:08.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:08 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:08.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:08 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:08.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:08 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:55:08.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:08 smithi053 ceph-mon[67180]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-02T04:55:08.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:08 smithi053 ceph-mon[67180]: mds.? [v2:172.21.15.122:6832/481906381,v1:172.21.15.122:6833/481906381] up:boot 2024-04-02T04:55:08.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:08 smithi053 ceph-mon[67180]: fsmap foofs:1 {0=foofs.smithi053.nfrfot=up:active} 1 up:standby 2024-04-02T04:55:08.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:08 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi122.mwoyoh"}]: dispatch 2024-04-02T04:55:09.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:09 smithi122 ceph-mon[53211]: pgmap v311: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 2.1 MiB/s wr, 101 op/s 2024-04-02T04:55:09.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:09 smithi053 ceph-mon[67180]: pgmap v311: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 2.1 MiB/s wr, 101 op/s 2024-04-02T04:55:11.810 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:11 smithi053 ceph-mon[67180]: pgmap v312: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 513 KiB/s wr, 76 op/s 2024-04-02T04:55:11.810 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:11 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:11.810 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:11 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:11.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:11 smithi122 ceph-mon[53211]: pgmap v312: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 513 KiB/s wr, 76 op/s 2024-04-02T04:55:11.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:11 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:11.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:11 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:12.593 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:12 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:55:12.593 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:12 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:12.593 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:12 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:12.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:12 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:55:12.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:12 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:12.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:12 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:13.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:13 smithi053 ceph-mon[67180]: pgmap v313: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 22 KiB/s wr, 37 op/s 2024-04-02T04:55:13.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:13 smithi122 ceph-mon[53211]: pgmap v313: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 22 KiB/s wr, 37 op/s 2024-04-02T04:55:15.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:15 smithi053 ceph-mon[67180]: pgmap v314: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 9.2 KiB/s wr, 28 op/s 2024-04-02T04:55:15.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:15 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:15.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:15 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:15.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:15 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:55:15.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:15 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:55:15.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:15 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:15.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:15 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:15.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:15 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:55:15.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:15 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:15.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:15 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:15.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:15 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:15.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:15 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:15.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:15 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:15.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:15 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:15.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:15 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi053.nfrfot"}]: dispatch 2024-04-02T04:55:15.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:15 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi053.nfrfot"}]': finished 2024-04-02T04:55:15.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:15 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi122.mwoyoh"}]: dispatch 2024-04-02T04:55:15.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:15 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi122.mwoyoh"}]': finished 2024-04-02T04:55:15.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:15 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:15.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:15 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:15.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:15 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:15.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:15 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:15.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:15 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:15.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:15 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:15.810 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:15 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:15.810 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:15 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:15.810 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:15 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:15.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:15 smithi122 ceph-mon[53211]: pgmap v314: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 9.2 KiB/s wr, 28 op/s 2024-04-02T04:55:15.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:15 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:15.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:15 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:15.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:15 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:55:15.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:15 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:55:15.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:15 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:15.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:15 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:15.897 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:15 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:55:15.897 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:15 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:15.897 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:15 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:15.897 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:15 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:15.897 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:15 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:15.897 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:15 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:15.897 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:15 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:15.897 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:15 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi053.nfrfot"}]: dispatch 2024-04-02T04:55:15.897 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:15 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi053.nfrfot"}]': finished 2024-04-02T04:55:15.898 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:15 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi122.mwoyoh"}]: dispatch 2024-04-02T04:55:15.898 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:15 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi122.mwoyoh"}]': finished 2024-04-02T04:55:15.898 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:15 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:15.898 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:15 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:15.898 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:15 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:15.898 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:15 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:15.898 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:15 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:15.899 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:15 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:15.899 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:15 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:15.899 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:15 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:15.899 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:15 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:16.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:16 smithi053 ceph-mon[67180]: Detected new or changed devices on smithi122 2024-04-02T04:55:16.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:16 smithi053 ceph-mon[67180]: pgmap v315: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 9.7 KiB/s wr, 11 op/s 2024-04-02T04:55:16.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:16 smithi053 ceph-mon[67180]: pgmap v316: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 4.5 MiB/s rd, 12 KiB/s wr, 8 op/s 2024-04-02T04:55:16.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:16 smithi053 ceph-mon[67180]: Upgrade: Setting container_image for all mds 2024-04-02T04:55:16.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:16 smithi053 ceph-mon[67180]: Upgrade: Setting container_image for all rgw 2024-04-02T04:55:16.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:16 smithi053 ceph-mon[67180]: Upgrade: Setting container_image for all rbd-mirror 2024-04-02T04:55:16.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:16 smithi053 ceph-mon[67180]: Upgrade: Setting container_image for all ceph-exporter 2024-04-02T04:55:16.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:16 smithi053 ceph-mon[67180]: Upgrade: Setting container_image for all iscsi 2024-04-02T04:55:16.808 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:16 smithi122 ceph-mon[53211]: Detected new or changed devices on smithi122 2024-04-02T04:55:16.808 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:16 smithi122 ceph-mon[53211]: pgmap v315: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 9.7 KiB/s wr, 11 op/s 2024-04-02T04:55:16.808 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:16 smithi122 ceph-mon[53211]: pgmap v316: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 4.5 MiB/s rd, 12 KiB/s wr, 8 op/s 2024-04-02T04:55:16.809 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:16 smithi122 ceph-mon[53211]: Upgrade: Setting container_image for all mds 2024-04-02T04:55:16.809 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:16 smithi122 ceph-mon[53211]: Upgrade: Setting container_image for all rgw 2024-04-02T04:55:16.809 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:16 smithi122 ceph-mon[53211]: Upgrade: Setting container_image for all rbd-mirror 2024-04-02T04:55:16.809 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:16 smithi122 ceph-mon[53211]: Upgrade: Setting container_image for all ceph-exporter 2024-04-02T04:55:16.809 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:16 smithi122 ceph-mon[53211]: Upgrade: Setting container_image for all iscsi 2024-04-02T04:55:18.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:18 smithi053 ceph-mon[67180]: pgmap v317: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 11 KiB/s wr, 4 op/s 2024-04-02T04:55:18.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:18 smithi053 ceph-mon[67180]: Upgrade: Updating nfs.foo.0.0.smithi053.mhkdcb (1/2) 2024-04-02T04:55:18.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:18 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:18.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:18 smithi053 ceph-mon[67180]: Creating key for client.nfs.foo.0.0.smithi053.mhkdcb 2024-04-02T04:55:18.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:18 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi053.mhkdcb", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-02T04:55:18.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:18 smithi053 ceph-mon[67180]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-04-02T04:55:18.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:18 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-02T04:55:18.559 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:18 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-02T04:55:18.559 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:18 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:55:18.559 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:18 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-02T04:55:18.559 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:18 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-02T04:55:18.559 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:18 smithi053 ceph-mon[67180]: Rados config object exists: conf-nfs.foo 2024-04-02T04:55:18.559 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:18 smithi053 ceph-mon[67180]: Creating key for client.nfs.foo.0.0.smithi053.mhkdcb-rgw 2024-04-02T04:55:18.559 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:18 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi053.mhkdcb-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-02T04:55:18.559 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:18 smithi053 ceph-mon[67180]: Bind address in nfs.foo.0.0.smithi053.mhkdcb's ganesha conf is defaulting to empty 2024-04-02T04:55:18.559 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:18 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:55:18.559 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:18 smithi053 ceph-mon[67180]: Deploying daemon nfs.foo.0.0.smithi053.mhkdcb on smithi053 2024-04-02T04:55:18.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:18 smithi122 ceph-mon[53211]: pgmap v317: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 11 KiB/s wr, 4 op/s 2024-04-02T04:55:18.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:18 smithi122 ceph-mon[53211]: Upgrade: Updating nfs.foo.0.0.smithi053.mhkdcb (1/2) 2024-04-02T04:55:18.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:18 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:18.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:18 smithi122 ceph-mon[53211]: Creating key for client.nfs.foo.0.0.smithi053.mhkdcb 2024-04-02T04:55:18.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:18 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi053.mhkdcb", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-02T04:55:18.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:18 smithi122 ceph-mon[53211]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-04-02T04:55:18.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:18 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-02T04:55:18.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:18 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-02T04:55:18.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:18 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:55:18.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:18 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-02T04:55:18.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:18 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-02T04:55:18.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:18 smithi122 ceph-mon[53211]: Rados config object exists: conf-nfs.foo 2024-04-02T04:55:18.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:18 smithi122 ceph-mon[53211]: Creating key for client.nfs.foo.0.0.smithi053.mhkdcb-rgw 2024-04-02T04:55:18.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:18 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi053.mhkdcb-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-02T04:55:18.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:18 smithi122 ceph-mon[53211]: Bind address in nfs.foo.0.0.smithi053.mhkdcb's ganesha conf is defaulting to empty 2024-04-02T04:55:18.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:18 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:55:18.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:18 smithi122 ceph-mon[53211]: Deploying daemon nfs.foo.0.0.smithi053.mhkdcb on smithi053 2024-04-02T04:55:20.535 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:20 smithi053 ceph-mon[67180]: pgmap v318: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 459 B/s rd, 11 KiB/s wr, 4 op/s 2024-04-02T04:55:20.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:20 smithi122 ceph-mon[53211]: pgmap v318: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 459 B/s rd, 11 KiB/s wr, 4 op/s 2024-04-02T04:55:22.477 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:22 smithi122 ceph-mon[53211]: pgmap v319: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 459 B/s rd, 229 B/s wr, 0 op/s 2024-04-02T04:55:22.477 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:22 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:22.477 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:22 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:22.477 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:22 smithi122 ceph-mon[53211]: from='mds.? [v2:172.21.15.53:6834/2789998387,v1:172.21.15.53:6835/2789998387]' entity='mds.foofs.smithi053.nfrfot' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.53:0/2653381202"}]: dispatch 2024-04-02T04:55:22.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:22 smithi053 ceph-mon[67180]: pgmap v319: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 459 B/s rd, 229 B/s wr, 0 op/s 2024-04-02T04:55:22.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:22 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:22.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:22 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:22.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:22 smithi053 ceph-mon[67180]: from='mds.? [v2:172.21.15.53:6834/2789998387,v1:172.21.15.53:6835/2789998387]' entity='mds.foofs.smithi053.nfrfot' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.53:0/2653381202"}]: dispatch 2024-04-02T04:55:23.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:23 smithi053 ceph-mon[67180]: Evicting (and blocklisting) client session 14538 (172.21.15.53:0/2653381202) 2024-04-02T04:55:23.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:23 smithi053 ceph-mon[67180]: from='mds.? [v2:172.21.15.53:6834/2789998387,v1:172.21.15.53:6835/2789998387]' entity='mds.foofs.smithi053.nfrfot' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.53:0/2653381202"}]': finished 2024-04-02T04:55:23.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:23 smithi053 ceph-mon[67180]: osdmap e151: 8 total, 8 up, 8 in 2024-04-02T04:55:23.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:23 smithi122 ceph-mon[53211]: Evicting (and blocklisting) client session 14538 (172.21.15.53:0/2653381202) 2024-04-02T04:55:23.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:23 smithi122 ceph-mon[53211]: from='mds.? [v2:172.21.15.53:6834/2789998387,v1:172.21.15.53:6835/2789998387]' entity='mds.foofs.smithi053.nfrfot' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.53:0/2653381202"}]': finished 2024-04-02T04:55:23.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:23 smithi122 ceph-mon[53211]: osdmap e151: 8 total, 8 up, 8 in 2024-04-02T04:55:24.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:24 smithi122 ceph-mon[53211]: pgmap v321: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 383 B/s wr, 0 op/s 2024-04-02T04:55:24.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:24 smithi122 ceph-mon[53211]: Upgrade: Updating nfs.foo.1.0.smithi122.mauvsb (2/2) 2024-04-02T04:55:24.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:24 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:24.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:24 smithi122 ceph-mon[53211]: Creating key for client.nfs.foo.1.0.smithi122.mauvsb 2024-04-02T04:55:24.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:24 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi122.mauvsb", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-02T04:55:24.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:24 smithi122 ceph-mon[53211]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-04-02T04:55:24.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:24 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-02T04:55:24.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:24 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-02T04:55:24.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:24 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:55:24.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:24 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-02T04:55:24.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:24 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-02T04:55:24.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:24 smithi122 ceph-mon[53211]: Rados config object exists: conf-nfs.foo 2024-04-02T04:55:24.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:24 smithi122 ceph-mon[53211]: Creating key for client.nfs.foo.1.0.smithi122.mauvsb-rgw 2024-04-02T04:55:24.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:24 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi122.mauvsb-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-02T04:55:24.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:24 smithi122 ceph-mon[53211]: Bind address in nfs.foo.1.0.smithi122.mauvsb's ganesha conf is defaulting to empty 2024-04-02T04:55:24.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:24 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:55:24.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:24 smithi122 ceph-mon[53211]: Deploying daemon nfs.foo.1.0.smithi122.mauvsb on smithi122 2024-04-02T04:55:24.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:24 smithi053 ceph-mon[67180]: pgmap v321: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 383 B/s wr, 0 op/s 2024-04-02T04:55:24.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:24 smithi053 ceph-mon[67180]: Upgrade: Updating nfs.foo.1.0.smithi122.mauvsb (2/2) 2024-04-02T04:55:24.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:24 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:24.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:24 smithi053 ceph-mon[67180]: Creating key for client.nfs.foo.1.0.smithi122.mauvsb 2024-04-02T04:55:24.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:24 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi122.mauvsb", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-02T04:55:24.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:24 smithi053 ceph-mon[67180]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-04-02T04:55:24.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:24 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-02T04:55:24.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:24 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-02T04:55:24.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:24 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:55:24.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:24 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-02T04:55:24.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:24 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-02T04:55:24.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:24 smithi053 ceph-mon[67180]: Rados config object exists: conf-nfs.foo 2024-04-02T04:55:24.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:24 smithi053 ceph-mon[67180]: Creating key for client.nfs.foo.1.0.smithi122.mauvsb-rgw 2024-04-02T04:55:24.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:24 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi122.mauvsb-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-02T04:55:24.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:24 smithi053 ceph-mon[67180]: Bind address in nfs.foo.1.0.smithi122.mauvsb's ganesha conf is defaulting to empty 2024-04-02T04:55:24.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:24 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:55:24.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:24 smithi053 ceph-mon[67180]: Deploying daemon nfs.foo.1.0.smithi122.mauvsb on smithi122 2024-04-02T04:55:26.515 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:26 smithi122 ceph-mon[53211]: pgmap v322: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 921 B/s wr, 3 op/s 2024-04-02T04:55:26.746 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:26 smithi053 ceph-mon[67180]: pgmap v322: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 921 B/s wr, 3 op/s 2024-04-02T04:55:27.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:27 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:55:27.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:27 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:27.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:27 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:27.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:27 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:55:27.725 INFO:teuthology.orchestra.run.smithi053.stdout:true 2024-04-02T04:55:27.725 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:27 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:55:27.725 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:27 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:27.726 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:27 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:27.726 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:27 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:55:28.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:28 smithi122 ceph-mon[53211]: pgmap v323: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-04-02T04:55:28.721 INFO:teuthology.orchestra.run.smithi053.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-02T04:55:28.721 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager.smithi053 smithi053 *:9093,9094 running (10m) 37s ago 19m 23.8M - 0.23.0 ba2b418f427c bf50c1cd2b3d 2024-04-02T04:55:28.721 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi053 smithi053 running (6m) 37s ago 18m 6660k - 19.0.0-1979-g6cec793d 78f830190839 4ce8915e0157 2024-04-02T04:55:28.722 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi122 smithi122 running (6m) 18s ago 18m 6660k - 19.0.0-1979-g6cec793d 78f830190839 e2b5369e6fdf 2024-04-02T04:55:28.722 INFO:teuthology.orchestra.run.smithi053.stdout:grafana.smithi053 smithi053 *:3000 error 37s ago 18m - - 2024-04-02T04:55:28.722 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi053.nfrfot smithi053 running (45s) 37s ago 13m 14.8M - 19.0.0-1979-g6cec793d 78f830190839 e7c0900d609e 2024-04-02T04:55:28.722 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi122.mwoyoh smithi122 running (21s) 18s ago 13m 10.4M - 19.0.0-1979-g6cec793d 78f830190839 e0f9c7c343dd 2024-04-02T04:55:28.722 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi053.legdmv smithi053 *:8443,9283,8765 running (9m) 37s ago 20m 511M - 19.0.0-1979-g6cec793d 78f830190839 c12d516a31bc 2024-04-02T04:55:28.722 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi122.cpwlyk smithi122 *:8443,9283,8765 running (8m) 18s ago 18m 452M - 19.0.0-1979-g6cec793d 78f830190839 9a545c5e7d81 2024-04-02T04:55:28.722 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi053 smithi053 running (7m) 37s ago 20m 59.5M 2048M 19.0.0-1979-g6cec793d 78f830190839 f53a44f19c00 2024-04-02T04:55:28.722 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi122 smithi122 running (7m) 18s ago 18m 45.6M 2048M 19.0.0-1979-g6cec793d 78f830190839 44b200fed3c5 2024-04-02T04:55:28.722 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.0.0.smithi053.mhkdcb smithi053 *:2049 starting - - - - 2024-04-02T04:55:28.722 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.1.0.smithi122.mauvsb smithi122 *:2049 starting - - - - 2024-04-02T04:55:28.722 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi053 smithi053 *:9100 running (9m) 37s ago 18m 18.1M - 1.3.1 1dbe0e931976 773b71971e67 2024-04-02T04:55:28.722 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi122 smithi122 *:9100 running (9m) 18s ago 18m 16.5M - 1.3.1 1dbe0e931976 475d439c7980 2024-04-02T04:55:28.722 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 smithi053 running (5m) 37s ago 17m 391M 1937M 19.0.0-1979-g6cec793d 78f830190839 490c346c7639 2024-04-02T04:55:28.723 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 smithi053 running (5m) 37s ago 17m 377M 1937M 19.0.0-1979-g6cec793d 78f830190839 e70e40949c2f 2024-04-02T04:55:28.723 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 smithi053 running (4m) 37s ago 16m 365M 1937M 19.0.0-1979-g6cec793d 78f830190839 4f053827e0dd 2024-04-02T04:55:28.723 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 smithi053 running (3m) 37s ago 16m 411M 1937M 19.0.0-1979-g6cec793d 78f830190839 5cd6de7a6535 2024-04-02T04:55:28.723 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 smithi122 running (3m) 18s ago 15m 365M 2704M 19.0.0-1979-g6cec793d 78f830190839 40d6ab1aac4b 2024-04-02T04:55:28.723 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 smithi122 running (2m) 18s ago 15m 332M 2704M 19.0.0-1979-g6cec793d 78f830190839 be26f425f51b 2024-04-02T04:55:28.723 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 smithi122 running (112s) 18s ago 15m 312M 2704M 19.0.0-1979-g6cec793d 78f830190839 aab18af876da 2024-04-02T04:55:28.723 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 smithi122 running (74s) 18s ago 14m 334M 2704M 19.0.0-1979-g6cec793d 78f830190839 86ee236f9426 2024-04-02T04:55:28.723 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus.smithi053 smithi053 *:9095 running (8m) 37s ago 17m 71.9M - 2.33.4 514e6a882f6e 1dd55399da10 2024-04-02T04:55:28.723 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:28 smithi053 ceph-mon[67180]: pgmap v323: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-04-02T04:55:29.146 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:55:29.147 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": { 2024-04-02T04:55:29.147 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:55:29.147 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:55:29.147 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": { 2024-04-02T04:55:29.147 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:55:29.147 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:55:29.147 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": { 2024-04-02T04:55:29.147 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 8 2024-04-02T04:55:29.147 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:55:29.147 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": { 2024-04-02T04:55:29.147 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:55:29.147 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:55:29.147 INFO:teuthology.orchestra.run.smithi053.stdout: "overall": { 2024-04-02T04:55:29.148 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 14 2024-04-02T04:55:29.148 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-02T04:55:29.148 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:55:29.537 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:55:29.537 INFO:teuthology.orchestra.run.smithi053.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae", 2024-04-02T04:55:29.537 INFO:teuthology.orchestra.run.smithi053.stdout: "in_progress": true, 2024-04-02T04:55:29.537 INFO:teuthology.orchestra.run.smithi053.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-02T04:55:29.537 INFO:teuthology.orchestra.run.smithi053.stdout: "services_complete": [ 2024-04-02T04:55:29.537 INFO:teuthology.orchestra.run.smithi053.stdout: "mon", 2024-04-02T04:55:29.537 INFO:teuthology.orchestra.run.smithi053.stdout: "crash", 2024-04-02T04:55:29.537 INFO:teuthology.orchestra.run.smithi053.stdout: "mds", 2024-04-02T04:55:29.537 INFO:teuthology.orchestra.run.smithi053.stdout: "osd", 2024-04-02T04:55:29.537 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr" 2024-04-02T04:55:29.537 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-04-02T04:55:29.537 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": "16/23 daemons upgraded", 2024-04-02T04:55:29.538 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "Currently upgrading nfs daemons", 2024-04-02T04:55:29.538 INFO:teuthology.orchestra.run.smithi053.stdout: "is_paused": false 2024-04-02T04:55:29.538 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:55:29.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:29 smithi122 ceph-mon[53211]: from='client.34502 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:55:29.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:29 smithi122 ceph-mon[53211]: from='client.34506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:55:29.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:29 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/2164472669' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:29.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:29 smithi053 ceph-mon[67180]: from='client.34502 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:55:29.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:29 smithi053 ceph-mon[67180]: from='client.34506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:55:29.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:29 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/2164472669' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:30.025 INFO:teuthology.orchestra.run.smithi053.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-02T04:55:30.025 INFO:teuthology.orchestra.run.smithi053.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-02T04:55:30.025 INFO:teuthology.orchestra.run.smithi053.stdout: daemon grafana.smithi053 on smithi053 is in error state 2024-04-02T04:55:30.769 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:30 smithi053 ceph-mon[67180]: from='client.34510 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:55:30.769 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:30 smithi053 ceph-mon[67180]: pgmap v324: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 4.6 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-02T04:55:30.769 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:30 smithi053 ceph-mon[67180]: from='client.34518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:55:30.769 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:30 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/395281428' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:55:30.769 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:30 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:30.770 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:30 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:30.770 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:30 smithi053 ceph-mon[67180]: Evicting (and blocklisting) client session 24369 (172.21.15.122:0/514971600) 2024-04-02T04:55:30.770 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:30 smithi053 ceph-mon[67180]: from='mds.? [v2:172.21.15.53:6834/2789998387,v1:172.21.15.53:6835/2789998387]' entity='mds.foofs.smithi053.nfrfot' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.122:0/514971600"}]: dispatch 2024-04-02T04:55:30.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:30 smithi122 ceph-mon[53211]: from='client.34510 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:55:30.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:30 smithi122 ceph-mon[53211]: pgmap v324: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 4.6 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-02T04:55:30.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:30 smithi122 ceph-mon[53211]: from='client.34518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:55:30.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:30 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/395281428' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:55:30.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:30 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:30.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:30 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:30.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:30 smithi122 ceph-mon[53211]: Evicting (and blocklisting) client session 24369 (172.21.15.122:0/514971600) 2024-04-02T04:55:30.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:30 smithi122 ceph-mon[53211]: from='mds.? [v2:172.21.15.53:6834/2789998387,v1:172.21.15.53:6835/2789998387]' entity='mds.foofs.smithi053.nfrfot' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.122:0/514971600"}]: dispatch 2024-04-02T04:55:32.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:32 smithi122 ceph-mon[53211]: pgmap v325: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 4.6 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-04-02T04:55:32.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:32 smithi122 ceph-mon[53211]: from='mds.? [v2:172.21.15.53:6834/2789998387,v1:172.21.15.53:6835/2789998387]' entity='mds.foofs.smithi053.nfrfot' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.122:0/514971600"}]': finished 2024-04-02T04:55:32.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:32 smithi122 ceph-mon[53211]: osdmap e152: 8 total, 8 up, 8 in 2024-04-02T04:55:32.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:32 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:32.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:32 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:32.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:32 smithi053 ceph-mon[67180]: pgmap v325: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 4.6 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-04-02T04:55:32.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:32 smithi053 ceph-mon[67180]: from='mds.? [v2:172.21.15.53:6834/2789998387,v1:172.21.15.53:6835/2789998387]' entity='mds.foofs.smithi053.nfrfot' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.122:0/514971600"}]': finished 2024-04-02T04:55:32.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:32 smithi053 ceph-mon[67180]: osdmap e152: 8 total, 8 up, 8 in 2024-04-02T04:55:32.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:32 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:32.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:32 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:33.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:33 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:33.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:33 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:33.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:33 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:33.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:33 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:34.517 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:34 smithi122 ceph-mon[53211]: pgmap v327: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-04-02T04:55:34.517 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:34 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:34.518 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:34 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:34.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:34 smithi053 ceph-mon[67180]: pgmap v327: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-04-02T04:55:34.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:34 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:34.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:34 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:35.712 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:35 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:35.712 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:35 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:35.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:35 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:35.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:35 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:36.610 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:36 smithi053 ceph-mon[67180]: pgmap v328: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-04-02T04:55:36.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:36 smithi122 ceph-mon[53211]: pgmap v328: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-04-02T04:55:38.346 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:38 smithi053 ceph-mon[67180]: pgmap v329: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-04-02T04:55:38.346 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:38 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:38.346 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:38 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:38.346 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:38 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:55:38.346 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:38 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:55:38.346 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:38 smithi053 ceph-mon[67180]: pgmap v330: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 244 B/s rd, 488 B/s wr, 1 op/s 2024-04-02T04:55:38.346 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:38 smithi053 ceph-mon[67180]: pgmap v331: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 320 B/s rd, 481 B/s wr, 1 op/s 2024-04-02T04:55:38.346 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:38 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:38.346 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:38 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:38.346 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:38 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:55:38.346 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:38 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:38.347 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:38 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:38.347 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:38 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:38.347 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:38 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:38.347 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:38 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:38.347 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:38 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:38.347 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:38 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:38.347 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:38 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:38.347 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:38 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:38.347 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:38 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:38.347 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:38 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:38.347 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:38 smithi053 ceph-mon[67180]: Upgrade: Setting container_image for all nfs 2024-04-02T04:55:38.347 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:38 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:38.347 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:38 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi053.mhkdcb"}]: dispatch 2024-04-02T04:55:38.347 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:38 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi053.mhkdcb"}]': finished 2024-04-02T04:55:38.348 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:38 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi122.mauvsb"}]: dispatch 2024-04-02T04:55:38.348 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:38 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi122.mauvsb"}]': finished 2024-04-02T04:55:38.348 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:38 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:38.348 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:38 smithi053 ceph-mon[67180]: Upgrade: Setting container_image for all nvmeof 2024-04-02T04:55:38.348 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:38 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:38.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:38 smithi122 ceph-mon[53211]: pgmap v329: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-04-02T04:55:38.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:38 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:38.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:38 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:38.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:38 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:55:38.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:38 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:55:38.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:38 smithi122 ceph-mon[53211]: pgmap v330: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 244 B/s rd, 488 B/s wr, 1 op/s 2024-04-02T04:55:38.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:38 smithi122 ceph-mon[53211]: pgmap v331: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 320 B/s rd, 481 B/s wr, 1 op/s 2024-04-02T04:55:38.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:38 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:38.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:38 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:38.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:38 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:55:38.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:38 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:38.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:38 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:38.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:38 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:38.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:38 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:38.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:38 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:38.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:38 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:38.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:38 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:38.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:38 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:38.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:38 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:38.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:38 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:38.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:38 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:38.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:38 smithi122 ceph-mon[53211]: Upgrade: Setting container_image for all nfs 2024-04-02T04:55:38.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:38 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:38.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:38 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi053.mhkdcb"}]: dispatch 2024-04-02T04:55:38.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:38 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi053.mhkdcb"}]': finished 2024-04-02T04:55:38.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:38 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi122.mauvsb"}]: dispatch 2024-04-02T04:55:38.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:38 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi122.mauvsb"}]': finished 2024-04-02T04:55:38.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:38 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:55:38.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:38 smithi122 ceph-mon[53211]: Upgrade: Setting container_image for all nvmeof 2024-04-02T04:55:38.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:38 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:40.438 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:40 smithi053 ceph-mon[67180]: Upgrade: Updating node-exporter.smithi053 (1/2) 2024-04-02T04:55:40.438 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:40 smithi053 ceph-mon[67180]: Deploying daemon node-exporter.smithi053 on smithi053 2024-04-02T04:55:40.438 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:40 smithi053 ceph-mon[67180]: pgmap v332: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 250 B/s rd, 375 B/s wr, 0 op/s 2024-04-02T04:55:40.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:40 smithi122 ceph-mon[53211]: Upgrade: Updating node-exporter.smithi053 (1/2) 2024-04-02T04:55:40.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:40 smithi122 ceph-mon[53211]: Deploying daemon node-exporter.smithi053 on smithi053 2024-04-02T04:55:40.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:40 smithi122 ceph-mon[53211]: pgmap v332: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 250 B/s rd, 375 B/s wr, 0 op/s 2024-04-02T04:55:42.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:42 smithi122 ceph-mon[53211]: pgmap v333: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 122 B/s rd, 366 B/s wr, 0 op/s 2024-04-02T04:55:42.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:42 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:55:42.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:42 smithi053 ceph-mon[67180]: pgmap v333: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 122 B/s rd, 366 B/s wr, 0 op/s 2024-04-02T04:55:42.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:42 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:55:44.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:44 smithi122 ceph-mon[53211]: pgmap v334: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 366 B/s wr, 0 op/s 2024-04-02T04:55:44.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:44 smithi053 ceph-mon[67180]: pgmap v334: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 366 B/s wr, 0 op/s 2024-04-02T04:55:45.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:45 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:45.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:45 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:46.060 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:45 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:46.060 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:45 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:46.656 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:46 smithi122 ceph-mon[53211]: pgmap v335: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 244 B/s rd, 0 op/s 2024-04-02T04:55:47.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:46 smithi053 ceph-mon[67180]: pgmap v335: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 244 B/s rd, 0 op/s 2024-04-02T04:55:47.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:47 smithi122 ceph-mon[53211]: Upgrade: Updating node-exporter.smithi122 (2/2) 2024-04-02T04:55:47.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:47 smithi122 ceph-mon[53211]: Deploying daemon node-exporter.smithi122 on smithi122 2024-04-02T04:55:48.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:47 smithi053 ceph-mon[67180]: Upgrade: Updating node-exporter.smithi122 (2/2) 2024-04-02T04:55:48.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:47 smithi053 ceph-mon[67180]: Deploying daemon node-exporter.smithi122 on smithi122 2024-04-02T04:55:48.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:48 smithi122 ceph-mon[53211]: pgmap v336: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-02T04:55:49.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:48 smithi053 ceph-mon[67180]: pgmap v336: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-02T04:55:50.683 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:50 smithi122 ceph-mon[53211]: pgmap v337: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:55:51.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:50 smithi053 ceph-mon[67180]: pgmap v337: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:55:51.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:51 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:51.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:51 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:51.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:51 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:55:51.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:51 smithi122 ceph-mon[53211]: pgmap v338: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:55:52.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:51 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:52.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:51 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:52.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:51 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:55:52.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:51 smithi053 ceph-mon[67180]: pgmap v338: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:55:54.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:54 smithi122 ceph-mon[53211]: pgmap v339: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:55:54.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:54.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:54.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:54 smithi053 ceph-mon[67180]: pgmap v339: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:55:54.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:54.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:56.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:55 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:56.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:55 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:56.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:55 smithi053 ceph-mon[67180]: pgmap v340: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-02T04:55:56.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:55 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:56.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:55 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:56.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:55 smithi122 ceph-mon[53211]: pgmap v340: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-02T04:55:56.995 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:56 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:56.995 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:56 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:56.995 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:56 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:55:57.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:56 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:57.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:56 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:57.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:56 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:55:58.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:57 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:58.307 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:57 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:58.308 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:57 smithi053 ceph-mon[67180]: pgmap v341: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:55:58.324 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:57 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:58.324 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:57 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:58.324 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:57 smithi122 ceph-mon[53211]: pgmap v341: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:55:59.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:59 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:59.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:55:59 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:59.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:59 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:55:59.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:55:59 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:00.408 INFO:teuthology.orchestra.run.smithi053.stdout:true 2024-04-02T04:56:00.731 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:00 smithi122 ceph-mon[53211]: pgmap v342: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:56:00.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:00 smithi053 ceph-mon[67180]: pgmap v342: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:56:01.225 INFO:teuthology.orchestra.run.smithi053.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-02T04:56:01.225 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager.smithi053 smithi053 *:9093,9094 running (10m) 5s ago 19m 23.3M - 0.23.0 ba2b418f427c bf50c1cd2b3d 2024-04-02T04:56:01.225 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi053 smithi053 running (6m) 5s ago 19m 6660k - 19.0.0-1979-g6cec793d 78f830190839 4ce8915e0157 2024-04-02T04:56:01.226 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi122 smithi122 running (6m) 7s ago 18m 6660k - 19.0.0-1979-g6cec793d 78f830190839 e2b5369e6fdf 2024-04-02T04:56:01.226 INFO:teuthology.orchestra.run.smithi053.stdout:grafana.smithi053 smithi053 *:3000 error 5s ago 19m - - 2024-04-02T04:56:01.226 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi053.nfrfot smithi053 running (78s) 5s ago 13m 34.7M - 19.0.0-1979-g6cec793d 78f830190839 e7c0900d609e 2024-04-02T04:56:01.226 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi122.mwoyoh smithi122 running (54s) 7s ago 13m 11.7M - 19.0.0-1979-g6cec793d 78f830190839 e0f9c7c343dd 2024-04-02T04:56:01.226 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi053.legdmv smithi053 *:8443,9283,8765 running (9m) 5s ago 20m 513M - 19.0.0-1979-g6cec793d 78f830190839 c12d516a31bc 2024-04-02T04:56:01.226 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi122.cpwlyk smithi122 *:8443,9283,8765 running (8m) 7s ago 18m 452M - 19.0.0-1979-g6cec793d 78f830190839 9a545c5e7d81 2024-04-02T04:56:01.226 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi053 smithi053 running (8m) 5s ago 21m 62.2M 2048M 19.0.0-1979-g6cec793d 78f830190839 f53a44f19c00 2024-04-02T04:56:01.226 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi122 smithi122 running (7m) 7s ago 18m 47.7M 2048M 19.0.0-1979-g6cec793d 78f830190839 44b200fed3c5 2024-04-02T04:56:01.226 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.0.0.smithi053.mhkdcb smithi053 *:2049 running (40s) 5s ago 13m 62.0M - 5.7 78f830190839 f15a31775645 2024-04-02T04:56:01.226 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.1.0.smithi122.mauvsb smithi122 *:2049 running (34s) 7s ago 13m 61.3M - 5.7 78f830190839 ca76a1d4d51f 2024-04-02T04:56:01.226 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi053 smithi053 *:9100 running (17s) 5s ago 19m 8024k - 1.5.0 0da6a335fe13 084c6c7e704a 2024-04-02T04:56:01.226 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi122 smithi122 *:9100 running (10s) 7s ago 18m 3418k - 1.5.0 0da6a335fe13 fd09a64a7e43 2024-04-02T04:56:01.226 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 smithi053 running (6m) 5s ago 18m 420M 1937M 19.0.0-1979-g6cec793d 78f830190839 490c346c7639 2024-04-02T04:56:01.226 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 smithi053 running (5m) 5s ago 17m 390M 1937M 19.0.0-1979-g6cec793d 78f830190839 e70e40949c2f 2024-04-02T04:56:01.227 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 smithi053 running (4m) 5s ago 17m 374M 1937M 19.0.0-1979-g6cec793d 78f830190839 4f053827e0dd 2024-04-02T04:56:01.227 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 smithi053 running (4m) 5s ago 16m 431M 1937M 19.0.0-1979-g6cec793d 78f830190839 5cd6de7a6535 2024-04-02T04:56:01.227 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 smithi122 running (3m) 7s ago 16m 367M 2704M 19.0.0-1979-g6cec793d 78f830190839 40d6ab1aac4b 2024-04-02T04:56:01.227 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 smithi122 running (3m) 7s ago 15m 333M 2704M 19.0.0-1979-g6cec793d 78f830190839 be26f425f51b 2024-04-02T04:56:01.227 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 smithi122 running (2m) 7s ago 15m 311M 2704M 19.0.0-1979-g6cec793d 78f830190839 aab18af876da 2024-04-02T04:56:01.227 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 smithi122 running (107s) 7s ago 15m 334M 2704M 19.0.0-1979-g6cec793d 78f830190839 86ee236f9426 2024-04-02T04:56:01.227 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus.smithi053 smithi053 *:9095 running (8m) 5s ago 18m 71.9M - 2.33.4 514e6a882f6e 1dd55399da10 2024-04-02T04:56:01.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:01 smithi053 ceph-mon[67180]: from='client.34528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:56:01.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:01.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:01.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:56:01.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:56:01.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:01.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:01.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:56:01.559 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:01.559 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:01.559 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:01.559 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:01.559 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:01.559 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:01.559 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:01.559 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:01.559 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:01.559 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:01.559 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:01.559 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:01.559 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:01 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:01.665 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:56:01.665 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": { 2024-04-02T04:56:01.665 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:56:01.665 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:56:01.665 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": { 2024-04-02T04:56:01.665 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:56:01.665 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:56:01.665 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": { 2024-04-02T04:56:01.665 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 8 2024-04-02T04:56:01.665 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:56:01.665 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": { 2024-04-02T04:56:01.665 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:56:01.666 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:56:01.666 INFO:teuthology.orchestra.run.smithi053.stdout: "overall": { 2024-04-02T04:56:01.666 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 14 2024-04-02T04:56:01.666 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-02T04:56:01.666 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:56:01.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:01 smithi122 ceph-mon[53211]: from='client.34528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:56:01.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:01.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:01.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:56:01.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:56:01.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:01.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:01.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:56:01.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:01.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:01.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:01.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:01.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:01.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:01.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:01.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:01.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:01.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:01.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:01.897 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:01.897 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:01 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:02.044 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:56:02.044 INFO:teuthology.orchestra.run.smithi053.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae", 2024-04-02T04:56:02.044 INFO:teuthology.orchestra.run.smithi053.stdout: "in_progress": true, 2024-04-02T04:56:02.044 INFO:teuthology.orchestra.run.smithi053.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-02T04:56:02.044 INFO:teuthology.orchestra.run.smithi053.stdout: "services_complete": [ 2024-04-02T04:56:02.044 INFO:teuthology.orchestra.run.smithi053.stdout: "nfs", 2024-04-02T04:56:02.044 INFO:teuthology.orchestra.run.smithi053.stdout: "mon", 2024-04-02T04:56:02.045 INFO:teuthology.orchestra.run.smithi053.stdout: "crash", 2024-04-02T04:56:02.045 INFO:teuthology.orchestra.run.smithi053.stdout: "mds", 2024-04-02T04:56:02.045 INFO:teuthology.orchestra.run.smithi053.stdout: "osd", 2024-04-02T04:56:02.045 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr" 2024-04-02T04:56:02.045 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-04-02T04:56:02.045 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": "18/23 daemons upgraded", 2024-04-02T04:56:02.045 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "Currently upgrading prometheus daemons", 2024-04-02T04:56:02.045 INFO:teuthology.orchestra.run.smithi053.stdout: "is_paused": false 2024-04-02T04:56:02.045 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:56:02.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:02 smithi053 ceph-mon[67180]: from='client.34532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:56:02.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:02 smithi053 ceph-mon[67180]: pgmap v343: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2024-04-02T04:56:02.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:02 smithi053 ceph-mon[67180]: pgmap v344: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 210 B/s rd, 0 op/s 2024-04-02T04:56:02.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:02 smithi053 ceph-mon[67180]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:56:02.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:02 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/1214718748' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:02.620 INFO:teuthology.orchestra.run.smithi053.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-02T04:56:02.620 INFO:teuthology.orchestra.run.smithi053.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-02T04:56:02.620 INFO:teuthology.orchestra.run.smithi053.stdout: daemon grafana.smithi053 on smithi053 is in error state 2024-04-02T04:56:02.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:02 smithi122 ceph-mon[53211]: from='client.34532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:56:02.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:02 smithi122 ceph-mon[53211]: pgmap v343: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2024-04-02T04:56:02.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:02 smithi122 ceph-mon[53211]: pgmap v344: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 210 B/s rd, 0 op/s 2024-04-02T04:56:02.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:02 smithi122 ceph-mon[53211]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:56:02.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:02 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/1214718748' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:03.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:03 smithi053 ceph-mon[67180]: from='client.34544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:56:03.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:03 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/2898342930' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:56:03.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:03 smithi122 ceph-mon[53211]: from='client.34544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:56:03.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:03 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/2898342930' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:56:04.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:04 smithi053 ceph-mon[67180]: Upgrade: Updating prometheus.smithi053 2024-04-02T04:56:04.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:04 smithi053 ceph-mon[67180]: pgmap v345: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 210 B/s rd, 0 op/s 2024-04-02T04:56:04.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:04 smithi053 ceph-mon[67180]: Deploying daemon prometheus.smithi053 on smithi053 2024-04-02T04:56:04.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:04 smithi122 ceph-mon[53211]: Upgrade: Updating prometheus.smithi053 2024-04-02T04:56:04.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:04 smithi122 ceph-mon[53211]: pgmap v345: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 210 B/s rd, 0 op/s 2024-04-02T04:56:04.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:04 smithi122 ceph-mon[53211]: Deploying daemon prometheus.smithi053 on smithi053 2024-04-02T04:56:06.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:06 smithi053 ceph-mon[67180]: pgmap v346: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 210 B/s rd, 0 op/s 2024-04-02T04:56:06.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:06 smithi122 ceph-mon[53211]: pgmap v346: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 210 B/s rd, 0 op/s 2024-04-02T04:56:08.660 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:08 smithi053 ceph-mon[67180]: pgmap v347: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 210 B/s rd, 0 op/s 2024-04-02T04:56:08.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:08 smithi122 ceph-mon[53211]: pgmap v347: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 210 B/s rd, 0 op/s 2024-04-02T04:56:10.419 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:10 smithi053 ceph-mon[67180]: pgmap v348: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 210 B/s rd, 0 op/s 2024-04-02T04:56:10.419 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:10 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:10.419 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:10 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:10.419 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:10 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:56:10.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:10 smithi122 ceph-mon[53211]: pgmap v348: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 210 B/s rd, 0 op/s 2024-04-02T04:56:10.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:10 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:10.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:10 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:10.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:10 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:56:12.554 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:12 smithi053 ceph-mon[67180]: pgmap v349: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-02T04:56:12.554 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:12 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:56:12.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:12 smithi122 ceph-mon[53211]: pgmap v349: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-02T04:56:12.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:12 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:56:14.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:14 smithi053 ceph-mon[67180]: pgmap v350: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:56:14.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:14 smithi122 ceph-mon[53211]: pgmap v350: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:56:15.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:15 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:15.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:15 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:15.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:15 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:15.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:15 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:16.747 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:16 smithi053 ceph-mon[67180]: pgmap v351: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-02T04:56:16.748 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:16 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:16.748 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:16 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:16.796 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:16 smithi122 ceph-mon[53211]: pgmap v351: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-02T04:56:16.796 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:16 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:16.797 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:16 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:18.571 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:18 smithi053 ceph-mon[67180]: pgmap v352: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:56:18.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:18 smithi122 ceph-mon[53211]: pgmap v352: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:56:20.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:20 smithi122 ceph-mon[53211]: pgmap v353: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:56:20.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:20 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:20.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:20 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:20.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:20 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:56:20.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:20 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:56:20.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:20 smithi122 ceph-mon[53211]: pgmap v354: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2024-04-02T04:56:20.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:20 smithi122 ceph-mon[53211]: pgmap v355: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 246 B/s rd, 0 op/s 2024-04-02T04:56:20.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:20 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:20.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:20 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:20.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:20 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-02T04:56:20.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:20 smithi122 ceph-mon[53211]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-02T04:56:20.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:20 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:56:20.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:20 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:20.647 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:20 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:20.648 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:20 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:20.648 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:20 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:20.648 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:20 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:20.648 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:20 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:20.648 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:20 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:20.648 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:20 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:20.648 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:20 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:20.648 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:20 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:20.649 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:20 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:20.649 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:20 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:20.649 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:20 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:20.649 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:20 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:20.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:20 smithi053 ceph-mon[67180]: pgmap v353: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:56:20.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:20 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:20.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:20 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:20.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:20 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:56:20.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:20 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:56:20.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:20 smithi053 ceph-mon[67180]: pgmap v354: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2024-04-02T04:56:20.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:20 smithi053 ceph-mon[67180]: pgmap v355: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 246 B/s rd, 0 op/s 2024-04-02T04:56:20.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:20 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:20.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:20 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:20.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:20 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-02T04:56:20.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:20 smithi053 ceph-mon[67180]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-02T04:56:20.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:20 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:56:20.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:20 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:20.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:20 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:20.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:20 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:20.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:20 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:20.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:20 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:20.810 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:20 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:20.810 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:20 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:20.810 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:20 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:20.810 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:20 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:20.810 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:20 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:20.810 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:20 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:20.810 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:20 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:20.810 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:20 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:20.810 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:20 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:22.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:22 smithi053 ceph-mon[67180]: Upgrade: Updating alertmanager.smithi053 2024-04-02T04:56:22.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:22 smithi053 ceph-mon[67180]: Deploying daemon alertmanager.smithi053 on smithi053 2024-04-02T04:56:22.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:22 smithi053 ceph-mon[67180]: pgmap v356: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 246 B/s rd, 0 op/s 2024-04-02T04:56:22.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:22 smithi122 ceph-mon[53211]: Upgrade: Updating alertmanager.smithi053 2024-04-02T04:56:22.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:22 smithi122 ceph-mon[53211]: Deploying daemon alertmanager.smithi053 on smithi053 2024-04-02T04:56:22.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:22 smithi122 ceph-mon[53211]: pgmap v356: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 246 B/s rd, 0 op/s 2024-04-02T04:56:24.703 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:24 smithi053 ceph-mon[67180]: pgmap v357: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-04-02T04:56:24.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:24 smithi122 ceph-mon[53211]: pgmap v357: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-04-02T04:56:26.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:26 smithi053 ceph-mon[67180]: pgmap v358: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 246 B/s rd, 0 op/s 2024-04-02T04:56:26.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:26 smithi122 ceph-mon[53211]: pgmap v358: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 246 B/s rd, 0 op/s 2024-04-02T04:56:27.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:27 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:56:27.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:27 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:27.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:27 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:27.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:27 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:56:27.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:27 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:56:27.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:27 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:27.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:27 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:27.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:27 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:56:28.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:28 smithi053 ceph-mon[67180]: pgmap v359: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 246 B/s rd, 0 op/s 2024-04-02T04:56:28.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:28 smithi122 ceph-mon[53211]: pgmap v359: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 246 B/s rd, 0 op/s 2024-04-02T04:56:30.446 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:30 smithi053 ceph-mon[67180]: pgmap v360: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-02T04:56:30.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:30 smithi122 ceph-mon[53211]: pgmap v360: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-02T04:56:32.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:32 smithi053 ceph-mon[67180]: pgmap v361: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:56:32.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:32 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:32.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:32 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:32.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:32 smithi122 ceph-mon[53211]: pgmap v361: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:56:32.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:32 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:32.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:32 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:33.035 INFO:teuthology.orchestra.run.smithi053.stdout:true 2024-04-02T04:56:33.792 INFO:teuthology.orchestra.run.smithi053.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-02T04:56:33.793 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager.smithi053 smithi053 *:9093,9094 running (7s) 1s ago 20m 16.9M - 0.25.0 c8568f914cd2 15d2f21d8153 2024-04-02T04:56:33.793 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi053 smithi053 running (7m) 1s ago 20m 6660k - 19.0.0-1979-g6cec793d 78f830190839 4ce8915e0157 2024-04-02T04:56:33.793 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi122 smithi122 running (7m) 40s ago 19m 6660k - 19.0.0-1979-g6cec793d 78f830190839 e2b5369e6fdf 2024-04-02T04:56:33.793 INFO:teuthology.orchestra.run.smithi053.stdout:grafana.smithi053 smithi053 *:3000 error 1s ago 19m - - 2024-04-02T04:56:33.793 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi053.nfrfot smithi053 running (111s) 1s ago 14m 34.9M - 19.0.0-1979-g6cec793d 78f830190839 e7c0900d609e 2024-04-02T04:56:33.793 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi122.mwoyoh smithi122 running (87s) 40s ago 14m 11.7M - 19.0.0-1979-g6cec793d 78f830190839 e0f9c7c343dd 2024-04-02T04:56:33.793 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi053.legdmv smithi053 *:8443,9283,8765 running (10m) 1s ago 21m 515M - 19.0.0-1979-g6cec793d 78f830190839 c12d516a31bc 2024-04-02T04:56:33.793 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi122.cpwlyk smithi122 *:8443,9283,8765 running (9m) 40s ago 19m 452M - 19.0.0-1979-g6cec793d 78f830190839 9a545c5e7d81 2024-04-02T04:56:33.793 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi053 smithi053 running (8m) 1s ago 21m 62.5M 2048M 19.0.0-1979-g6cec793d 78f830190839 f53a44f19c00 2024-04-02T04:56:33.793 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi122 smithi122 running (8m) 40s ago 19m 47.7M 2048M 19.0.0-1979-g6cec793d 78f830190839 44b200fed3c5 2024-04-02T04:56:33.793 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.0.0.smithi053.mhkdcb smithi053 *:2049 running (72s) 1s ago 13m 62.3M - 5.7 78f830190839 f15a31775645 2024-04-02T04:56:33.793 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.1.0.smithi122.mauvsb smithi122 *:2049 running (67s) 40s ago 13m 61.3M - 5.7 78f830190839 ca76a1d4d51f 2024-04-02T04:56:33.793 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi053 smithi053 *:9100 running (49s) 1s ago 19m 8102k - 1.5.0 0da6a335fe13 084c6c7e704a 2024-04-02T04:56:33.794 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi122 smithi122 *:9100 running (43s) 40s ago 19m 3418k - 1.5.0 0da6a335fe13 fd09a64a7e43 2024-04-02T04:56:33.794 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 smithi053 running (6m) 1s ago 18m 421M 1937M 19.0.0-1979-g6cec793d 78f830190839 490c346c7639 2024-04-02T04:56:33.794 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 smithi053 running (6m) 1s ago 18m 391M 1937M 19.0.0-1979-g6cec793d 78f830190839 e70e40949c2f 2024-04-02T04:56:33.794 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 smithi053 running (5m) 1s ago 17m 376M 1937M 19.0.0-1979-g6cec793d 78f830190839 4f053827e0dd 2024-04-02T04:56:33.794 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 smithi053 running (4m) 1s ago 17m 432M 1937M 19.0.0-1979-g6cec793d 78f830190839 5cd6de7a6535 2024-04-02T04:56:33.794 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 smithi122 running (4m) 40s ago 16m 367M 2704M 19.0.0-1979-g6cec793d 78f830190839 40d6ab1aac4b 2024-04-02T04:56:33.794 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 smithi122 running (3m) 40s ago 16m 333M 2704M 19.0.0-1979-g6cec793d 78f830190839 be26f425f51b 2024-04-02T04:56:33.794 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 smithi122 running (2m) 40s ago 16m 311M 2704M 19.0.0-1979-g6cec793d 78f830190839 aab18af876da 2024-04-02T04:56:33.794 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 smithi122 running (2m) 40s ago 15m 334M 2704M 19.0.0-1979-g6cec793d 78f830190839 86ee236f9426 2024-04-02T04:56:33.794 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus.smithi053 smithi053 *:9095 running (24s) 1s ago 18m 44.7M - 2.43.0 a07b618ecd1d 4add6ebe92ce 2024-04-02T04:56:34.212 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:56:34.212 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": { 2024-04-02T04:56:34.212 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:56:34.212 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:56:34.212 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": { 2024-04-02T04:56:34.212 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:56:34.212 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:56:34.212 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": { 2024-04-02T04:56:34.212 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 8 2024-04-02T04:56:34.212 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:56:34.213 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": { 2024-04-02T04:56:34.213 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:56:34.213 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:56:34.213 INFO:teuthology.orchestra.run.smithi053.stdout: "overall": { 2024-04-02T04:56:34.213 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 14 2024-04-02T04:56:34.213 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-02T04:56:34.213 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:56:34.304 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:34 smithi053 ceph-mon[67180]: from='client.34552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:56:34.304 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:34 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:34.304 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:34 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:34.304 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:34 smithi053 ceph-mon[67180]: pgmap v362: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:56:34.304 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:34 smithi053 ceph-mon[67180]: from='client.34556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:56:34.304 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:34 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/3743460778' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:34.558 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:56:34.558 INFO:teuthology.orchestra.run.smithi053.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae", 2024-04-02T04:56:34.558 INFO:teuthology.orchestra.run.smithi053.stdout: "in_progress": true, 2024-04-02T04:56:34.558 INFO:teuthology.orchestra.run.smithi053.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-02T04:56:34.559 INFO:teuthology.orchestra.run.smithi053.stdout: "services_complete": [ 2024-04-02T04:56:34.559 INFO:teuthology.orchestra.run.smithi053.stdout: "nfs", 2024-04-02T04:56:34.559 INFO:teuthology.orchestra.run.smithi053.stdout: "mon", 2024-04-02T04:56:34.559 INFO:teuthology.orchestra.run.smithi053.stdout: "crash", 2024-04-02T04:56:34.559 INFO:teuthology.orchestra.run.smithi053.stdout: "mds", 2024-04-02T04:56:34.559 INFO:teuthology.orchestra.run.smithi053.stdout: "osd", 2024-04-02T04:56:34.559 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr" 2024-04-02T04:56:34.559 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-04-02T04:56:34.559 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": "18/23 daemons upgraded", 2024-04-02T04:56:34.559 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "Currently upgrading alertmanager daemons", 2024-04-02T04:56:34.559 INFO:teuthology.orchestra.run.smithi053.stdout: "is_paused": false 2024-04-02T04:56:34.559 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:56:34.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:34 smithi122 ceph-mon[53211]: from='client.34552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:56:34.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:34 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:34.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:34 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:34.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:34 smithi122 ceph-mon[53211]: pgmap v362: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:56:34.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:34 smithi122 ceph-mon[53211]: from='client.34556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:56:34.646 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:34 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/3743460778' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:34.965 INFO:teuthology.orchestra.run.smithi053.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-02T04:56:34.965 INFO:teuthology.orchestra.run.smithi053.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-02T04:56:34.965 INFO:teuthology.orchestra.run.smithi053.stdout: daemon grafana.smithi053 on smithi053 is in error state 2024-04-02T04:56:35.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:35 smithi053 ceph-mon[67180]: from='client.34560 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:56:35.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:35 smithi053 ceph-mon[67180]: from='client.34568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:56:35.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:35 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/2378246170' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:56:35.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:35 smithi122 ceph-mon[53211]: from='client.34560 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:56:35.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:35 smithi122 ceph-mon[53211]: from='client.34568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:56:35.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:35 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/2378246170' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:56:36.500 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:36 smithi053 ceph-mon[67180]: pgmap v363: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-02T04:56:36.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:36 smithi122 ceph-mon[53211]: pgmap v363: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-02T04:56:37.911 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:37 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:37.912 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:37 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:37.912 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:37 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:56:37.912 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:37 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:56:37.912 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:37 smithi053 ceph-mon[67180]: pgmap v364: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 178 B/s rd, 0 op/s 2024-04-02T04:56:37.912 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:37 smithi053 ceph-mon[67180]: pgmap v365: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 215 B/s rd, 0 op/s 2024-04-02T04:56:37.912 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:37 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:37.912 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:37 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:37.912 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:37 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-02T04:56:37.912 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:37 smithi053 ceph-mon[67180]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-02T04:56:37.912 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:37 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:56:37.912 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:37 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:37.912 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:37 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:37.912 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:37 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:37.912 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:37 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:37.913 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:37 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:37.913 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:37 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:37.913 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:37 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:37.913 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:37 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:37.913 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:37 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:37.913 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:37 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:37.913 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:37 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:37.913 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:37 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:37.913 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:37 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:37.913 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:37 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:37.913 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:37 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:38.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:37 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:38.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:37 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:38.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:37 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:56:38.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:37 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:56:38.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:37 smithi122 ceph-mon[53211]: pgmap v364: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 178 B/s rd, 0 op/s 2024-04-02T04:56:38.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:37 smithi122 ceph-mon[53211]: pgmap v365: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 215 B/s rd, 0 op/s 2024-04-02T04:56:38.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:37 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:38.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:37 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:38.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:37 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-02T04:56:38.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:37 smithi122 ceph-mon[53211]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-02T04:56:38.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:37 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:56:38.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:37 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:38.146 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:37 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:38.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:37 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:38.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:37 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:38.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:37 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:38.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:37 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:38.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:37 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:38.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:37 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:38.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:37 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:38.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:37 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:38.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:37 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:38.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:37 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:38.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:37 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:38.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:37 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:38.147 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:37 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:40.021 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:39 smithi053 ceph-mon[67180]: Upgrade: Updating grafana.smithi053 2024-04-02T04:56:40.021 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:39 smithi053 ceph-mon[67180]: Deploying daemon grafana.smithi053 on smithi053 2024-04-02T04:56:40.022 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:39 smithi053 ceph-mon[67180]: pgmap v366: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 215 B/s rd, 0 op/s 2024-04-02T04:56:40.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:39 smithi122 ceph-mon[53211]: Upgrade: Updating grafana.smithi053 2024-04-02T04:56:40.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:39 smithi122 ceph-mon[53211]: Deploying daemon grafana.smithi053 on smithi053 2024-04-02T04:56:40.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:39 smithi122 ceph-mon[53211]: pgmap v366: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 215 B/s rd, 0 op/s 2024-04-02T04:56:42.248 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:41 smithi053 ceph-mon[67180]: pgmap v367: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 215 B/s rd, 0 op/s 2024-04-02T04:56:42.248 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:41 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:56:42.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:41 smithi122 ceph-mon[53211]: pgmap v367: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 215 B/s rd, 0 op/s 2024-04-02T04:56:42.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:41 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:56:43.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:43 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:43.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:43 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:43.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:43 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:56:43.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:43 smithi122 ceph-mon[53211]: pgmap v368: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 215 B/s rd, 0 op/s 2024-04-02T04:56:43.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:43 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:43.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:43 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:44.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:43 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:44.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:43 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:44.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:43 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:56:44.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:43 smithi053 ceph-mon[67180]: pgmap v368: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 215 B/s rd, 0 op/s 2024-04-02T04:56:44.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:43 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:44.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:43 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:46.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:45 smithi053 ceph-mon[67180]: pgmap v369: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 215 B/s rd, 0 op/s 2024-04-02T04:56:46.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:45 smithi122 ceph-mon[53211]: pgmap v369: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 215 B/s rd, 0 op/s 2024-04-02T04:56:48.280 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:47 smithi053 ceph-mon[67180]: pgmap v370: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-02T04:56:48.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:47 smithi122 ceph-mon[53211]: pgmap v370: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-02T04:56:49.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:49 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:49.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:49 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:49.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:49 smithi122 ceph-mon[53211]: pgmap v371: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:56:50.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:49 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:50.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:49 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:50.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:49 smithi053 ceph-mon[67180]: pgmap v371: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:56:51.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:50 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:51.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:50 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:51.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:50 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:51.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:50 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:52.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:51 smithi053 ceph-mon[67180]: pgmap v372: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:56:52.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:51 smithi122 ceph-mon[53211]: pgmap v372: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:56:54.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: pgmap v373: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:56:54.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:54.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:54.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:56:54.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:56:54.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: pgmap v374: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-04-02T04:56:54.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:54.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:54.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-02T04:56:54.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:56:54.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.809 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.810 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.810 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.810 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.810 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.810 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.810 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.810 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.810 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.810 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.811 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.811 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:54.811 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-02T04:56:54.811 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-02T04:56:54.811 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-02T04:56:54.811 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-02T04:56:54.811 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-02T04:56:54.811 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-02T04:56:54.811 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-02T04:56:54.811 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-02T04:56:54.812 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-02T04:56:54.812 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-02T04:56:54.812 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-02T04:56:54.812 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-02T04:56:54.812 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-02T04:56:54.812 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-02T04:56:54.812 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-02T04:56:54.812 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-02T04:56:54.812 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-02T04:56:54.813 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-02T04:56:54.813 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-02T04:56:54.813 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-02T04:56:54.813 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-02T04:56:54.813 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-02T04:56:54.813 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-02T04:56:54.813 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-02T04:56:54.813 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-02T04:56:54.813 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-02T04:56:54.814 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-02T04:56:54.814 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-02T04:56:54.814 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-02T04:56:54.814 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-02T04:56:54.814 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-02T04:56:54.814 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:56:54.814 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:56:54.814 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:56:54.814 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:54.815 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:54.815 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:56:54.815 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:56:54.815 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:56:54.815 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:54.815 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:54.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: pgmap v373: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:56:54.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:54.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:54.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:56:54.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:56:54.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: pgmap v374: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-04-02T04:56:54.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:54.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:54.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-02T04:56:54.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:56:54.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.897 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.897 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.897 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.897 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.897 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.897 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.897 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.897 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.897 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.897 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.898 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.898 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.898 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.898 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.898 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:56:54.898 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:54.898 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-02T04:56:54.898 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-02T04:56:54.898 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-02T04:56:54.898 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-02T04:56:54.898 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-02T04:56:54.899 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-02T04:56:54.899 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-02T04:56:54.899 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-02T04:56:54.899 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-02T04:56:54.899 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-02T04:56:54.899 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-02T04:56:54.899 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-02T04:56:54.899 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-02T04:56:54.899 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-02T04:56:54.899 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-02T04:56:54.899 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-02T04:56:54.899 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-02T04:56:54.899 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-02T04:56:54.900 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-02T04:56:54.900 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-02T04:56:54.900 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-02T04:56:54.900 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-02T04:56:54.900 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-02T04:56:54.900 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-02T04:56:54.900 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-02T04:56:54.900 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-02T04:56:54.900 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-02T04:56:54.900 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-02T04:56:54.900 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-02T04:56:54.901 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-02T04:56:54.901 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-02T04:56:54.901 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:56:54.901 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:56:54.901 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:56:54.901 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:54.901 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:54.901 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:56:54.901 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:56:54.901 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:56:54.901 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:54.901 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:55.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:55 smithi053 ceph-mon[67180]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-02T04:56:55.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:55 smithi053 ceph-mon[67180]: Upgrade: Finalizing container_image settings 2024-04-02T04:56:55.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:55 smithi053 ceph-mon[67180]: Upgrade: Complete! 2024-04-02T04:56:55.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:55 smithi053 ceph-mon[67180]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-02T04:56:55.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:55 smithi053 ceph-mon[67180]: Cluster is now healthy 2024-04-02T04:56:55.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:55 smithi122 ceph-mon[53211]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-02T04:56:55.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:55 smithi122 ceph-mon[53211]: Upgrade: Finalizing container_image settings 2024-04-02T04:56:55.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:55 smithi122 ceph-mon[53211]: Upgrade: Complete! 2024-04-02T04:56:55.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:55 smithi122 ceph-mon[53211]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-02T04:56:55.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:55 smithi122 ceph-mon[53211]: Cluster is now healthy 2024-04-02T04:56:56.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:56 smithi053 ceph-mon[67180]: pgmap v375: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 191 B/s rd, 0 op/s 2024-04-02T04:56:56.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:56 smithi122 ceph-mon[53211]: pgmap v375: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 191 B/s rd, 0 op/s 2024-04-02T04:56:57.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:57 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:57.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:57 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:56:57.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:57 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:57.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:56:57 smithi122 ceph-mon[53211]: pgmap v376: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 191 B/s rd, 0 op/s 2024-04-02T04:56:58.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:57 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:58.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:57 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:56:58.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:57 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:56:58.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:56:57 smithi053 ceph-mon[67180]: pgmap v376: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 191 B/s rd, 0 op/s 2024-04-02T04:57:00.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:00 smithi122 ceph-mon[53211]: pgmap v377: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 191 B/s rd, 0 op/s 2024-04-02T04:57:01.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:00 smithi053 ceph-mon[67180]: pgmap v377: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 191 B/s rd, 0 op/s 2024-04-02T04:57:02.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:01 smithi053 ceph-mon[67180]: pgmap v378: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 191 B/s rd, 0 op/s 2024-04-02T04:57:02.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:01 smithi122 ceph-mon[53211]: pgmap v378: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 191 B/s rd, 0 op/s 2024-04-02T04:57:04.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:04 smithi122 ceph-mon[53211]: pgmap v379: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-04-02T04:57:05.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:04 smithi053 ceph-mon[67180]: pgmap v379: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-04-02T04:57:05.912 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -e sha1=6cec793d9073f0594e2364b9b687ecaf22424cca -- bash -c 'ceph orch ps' 2024-04-02T04:57:06.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:06 smithi053 ceph-mon[67180]: from='client.34576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:57:06.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:06 smithi053 ceph-mon[67180]: pgmap v380: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:06.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:06 smithi122 ceph-mon[53211]: from='client.34576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:57:06.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:06 smithi122 ceph-mon[53211]: pgmap v380: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:07.925 INFO:teuthology.orchestra.run.smithi053.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-02T04:57:07.925 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager.smithi053 smithi053 *:9093,9094 running (41s) 19s ago 20m 16.9M - 0.25.0 c8568f914cd2 15d2f21d8153 2024-04-02T04:57:07.925 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi053 smithi053 running (7m) 19s ago 20m 6660k - 19.0.0-1979-g6cec793d 78f830190839 4ce8915e0157 2024-04-02T04:57:07.925 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi122 smithi122 running (7m) 74s ago 20m 6660k - 19.0.0-1979-g6cec793d 78f830190839 e2b5369e6fdf 2024-04-02T04:57:07.925 INFO:teuthology.orchestra.run.smithi053.stdout:grafana.smithi053 smithi053 *:3000 running (25s) 19s ago 20m 61.0M - 9.4.12 2bacad6d85d8 6605ae1a36e4 2024-04-02T04:57:07.925 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi053.nfrfot smithi053 running (2m) 19s ago 14m 35.0M - 19.0.0-1979-g6cec793d 78f830190839 e7c0900d609e 2024-04-02T04:57:07.925 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi122.mwoyoh smithi122 running (2m) 74s ago 14m 11.7M - 19.0.0-1979-g6cec793d 78f830190839 e0f9c7c343dd 2024-04-02T04:57:07.925 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi053.legdmv smithi053 *:8443,9283,8765 running (10m) 19s ago 22m 521M - 19.0.0-1979-g6cec793d 78f830190839 c12d516a31bc 2024-04-02T04:57:07.925 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi122.cpwlyk smithi122 *:8443,9283,8765 running (9m) 74s ago 20m 452M - 19.0.0-1979-g6cec793d 78f830190839 9a545c5e7d81 2024-04-02T04:57:07.925 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi053 smithi053 running (9m) 19s ago 22m 63.2M 2048M 19.0.0-1979-g6cec793d 78f830190839 f53a44f19c00 2024-04-02T04:57:07.925 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi122 smithi122 running (8m) 74s ago 19m 47.7M 2048M 19.0.0-1979-g6cec793d 78f830190839 44b200fed3c5 2024-04-02T04:57:07.925 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.0.0.smithi053.mhkdcb smithi053 *:2049 running (106s) 19s ago 14m 63.7M - 5.7 78f830190839 f15a31775645 2024-04-02T04:57:07.925 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.1.0.smithi122.mauvsb smithi122 *:2049 running (101s) 74s ago 14m 61.3M - 5.7 78f830190839 ca76a1d4d51f 2024-04-02T04:57:07.926 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi053 smithi053 *:9100 running (83s) 19s ago 20m 7856k - 1.5.0 0da6a335fe13 084c6c7e704a 2024-04-02T04:57:07.926 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi122 smithi122 *:9100 running (77s) 74s ago 19m 3418k - 1.5.0 0da6a335fe13 fd09a64a7e43 2024-04-02T04:57:07.926 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 smithi053 running (7m) 19s ago 19m 422M 1937M 19.0.0-1979-g6cec793d 78f830190839 490c346c7639 2024-04-02T04:57:07.926 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 smithi053 running (6m) 19s ago 18m 392M 1937M 19.0.0-1979-g6cec793d 78f830190839 e70e40949c2f 2024-04-02T04:57:07.926 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 smithi053 running (6m) 19s ago 18m 376M 1937M 19.0.0-1979-g6cec793d 78f830190839 4f053827e0dd 2024-04-02T04:57:07.926 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 smithi053 running (5m) 19s ago 17m 432M 1937M 19.0.0-1979-g6cec793d 78f830190839 5cd6de7a6535 2024-04-02T04:57:07.926 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 smithi122 running (4m) 74s ago 17m 367M 2704M 19.0.0-1979-g6cec793d 78f830190839 40d6ab1aac4b 2024-04-02T04:57:07.926 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 smithi122 running (4m) 74s ago 17m 333M 2704M 19.0.0-1979-g6cec793d 78f830190839 be26f425f51b 2024-04-02T04:57:07.926 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 smithi122 running (3m) 74s ago 16m 311M 2704M 19.0.0-1979-g6cec793d 78f830190839 aab18af876da 2024-04-02T04:57:07.926 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 smithi122 running (2m) 74s ago 16m 334M 2704M 19.0.0-1979-g6cec793d 78f830190839 86ee236f9426 2024-04-02T04:57:07.926 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus.smithi053 smithi053 *:9095 running (59s) 19s ago 19m 44.9M - 2.43.0 a07b618ecd1d 4add6ebe92ce 2024-04-02T04:57:08.624 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -e sha1=6cec793d9073f0594e2364b9b687ecaf22424cca -- bash -c 'ceph versions' 2024-04-02T04:57:08.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:08 smithi122 ceph-mon[53211]: pgmap v381: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:08.954 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:08 smithi053 ceph-mon[67180]: pgmap v381: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:09.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:09 smithi122 ceph-mon[53211]: from='client.34580 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:57:09.902 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:09 smithi053 ceph-mon[67180]: from='client.34580 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:57:10.702 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:57:10.702 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": { 2024-04-02T04:57:10.702 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:57:10.702 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:57:10.702 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": { 2024-04-02T04:57:10.702 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:57:10.702 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:57:10.702 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": { 2024-04-02T04:57:10.702 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 8 2024-04-02T04:57:10.702 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:57:10.703 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": { 2024-04-02T04:57:10.703 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:57:10.703 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:57:10.703 INFO:teuthology.orchestra.run.smithi053.stdout: "overall": { 2024-04-02T04:57:10.703 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 14 2024-04-02T04:57:10.703 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-02T04:57:10.703 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:57:10.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:10 smithi122 ceph-mon[53211]: pgmap v382: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:11.026 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:10 smithi053 ceph-mon[67180]: pgmap v382: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:11.382 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -e sha1=6cec793d9073f0594e2364b9b687ecaf22424cca -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-04-02T04:57:11.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:11 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/4086466693' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:57:11.905 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:11 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/4086466693' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:57:12.875 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:12 smithi053 ceph-mon[67180]: pgmap v383: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:12.875 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:12 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:57:12.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:12 smithi122 ceph-mon[53211]: pgmap v383: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:12.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:12 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:57:13.087 INFO:teuthology.orchestra.run.smithi053.stdout:wait for servicemap items w/ changing names to refresh 2024-04-02T04:57:13.716 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:13 smithi053 ceph-mon[67180]: pgmap v384: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-02T04:57:13.739 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -e sha1=6cec793d9073f0594e2364b9b687ecaf22424cca -- bash -c 'sleep 60' 2024-04-02T04:57:13.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:13 smithi122 ceph-mon[53211]: pgmap v384: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-02T04:57:16.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:16 smithi122 ceph-mon[53211]: pgmap v385: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:17.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:16 smithi053 ceph-mon[67180]: pgmap v385: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:18.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:18 smithi122 ceph-mon[53211]: pgmap v386: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:19.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:18 smithi053 ceph-mon[67180]: pgmap v386: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:20.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:20 smithi122 ceph-mon[53211]: pgmap v387: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:21.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:20 smithi053 ceph-mon[67180]: pgmap v387: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:22.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:22 smithi122 ceph-mon[53211]: pgmap v388: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:23.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:22 smithi053 ceph-mon[67180]: pgmap v388: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:24.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:24 smithi122 ceph-mon[53211]: pgmap v389: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-02T04:57:25.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:24 smithi053 ceph-mon[67180]: pgmap v389: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-02T04:57:26.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:26 smithi122 ceph-mon[53211]: pgmap v390: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:27.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:26 smithi053 ceph-mon[67180]: pgmap v390: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:27.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:27 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:57:28.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:27 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:57:28.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:28 smithi122 ceph-mon[53211]: pgmap v391: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:29.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:28 smithi053 ceph-mon[67180]: pgmap v391: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:29.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:29 smithi122 ceph-mon[53211]: pgmap v392: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:30.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:29 smithi053 ceph-mon[67180]: pgmap v392: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:32.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:32 smithi122 ceph-mon[53211]: pgmap v393: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:33.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:32 smithi053 ceph-mon[67180]: pgmap v393: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:34.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:34 smithi122 ceph-mon[53211]: pgmap v394: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-02T04:57:35.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:34 smithi053 ceph-mon[67180]: pgmap v394: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-02T04:57:36.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:36 smithi122 ceph-mon[53211]: pgmap v395: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:37.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:36 smithi053 ceph-mon[67180]: pgmap v395: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:38.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:38 smithi122 ceph-mon[53211]: pgmap v396: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:39.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:38 smithi053 ceph-mon[67180]: pgmap v396: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:40.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:40 smithi122 ceph-mon[53211]: pgmap v397: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:41.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:40 smithi053 ceph-mon[67180]: pgmap v397: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:42.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:42 smithi122 ceph-mon[53211]: pgmap v398: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:42.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:42 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:57:43.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:42 smithi053 ceph-mon[67180]: pgmap v398: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:43.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:42 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:57:43.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:43 smithi122 ceph-mon[53211]: pgmap v399: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-02T04:57:44.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:43 smithi053 ceph-mon[67180]: pgmap v399: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-02T04:57:46.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:46 smithi122 ceph-mon[53211]: pgmap v400: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:47.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:46 smithi053 ceph-mon[67180]: pgmap v400: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:48.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:48 smithi122 ceph-mon[53211]: pgmap v401: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:49.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:48 smithi053 ceph-mon[67180]: pgmap v401: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:50.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:50 smithi122 ceph-mon[53211]: pgmap v402: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:51.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:50 smithi053 ceph-mon[67180]: pgmap v402: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:52.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:52 smithi122 ceph-mon[53211]: pgmap v403: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:53.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:52 smithi053 ceph-mon[67180]: pgmap v403: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:54.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:54 smithi122 ceph-mon[53211]: pgmap v404: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-02T04:57:54.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:57:54.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:57:54.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:54 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:57:55.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:54 smithi053 ceph-mon[67180]: pgmap v404: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-02T04:57:55.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T04:57:55.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:57:55.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:54 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:57:55.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:55 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:57:55.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:55 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:57:55.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:55 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:57:55.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:55 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:57:55.896 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:55 smithi122 ceph-mon[53211]: pgmap v405: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:56.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:55 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T04:57:56.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:55 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T04:57:56.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:55 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:57:56.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:55 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' 2024-04-02T04:57:56.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:55 smithi053 ceph-mon[67180]: pgmap v405: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:57.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:57 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:57:57.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:57 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:57:58.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:57:58 smithi122 ceph-mon[53211]: pgmap v406: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:57:58.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:57:58 smithi053 ceph-mon[67180]: pgmap v406: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:00.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:00 smithi122 ceph-mon[53211]: pgmap v407: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:01.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:00 smithi053 ceph-mon[67180]: pgmap v407: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:02.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:02 smithi122 ceph-mon[53211]: pgmap v408: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:03.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:02 smithi053 ceph-mon[67180]: pgmap v408: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:04.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:04 smithi122 ceph-mon[53211]: pgmap v409: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-02T04:58:05.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:04 smithi053 ceph-mon[67180]: pgmap v409: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-02T04:58:06.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:06 smithi122 ceph-mon[53211]: pgmap v410: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:07.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:06 smithi053 ceph-mon[67180]: pgmap v410: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:07.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:07 smithi122 ceph-mon[53211]: pgmap v411: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:08.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:07 smithi053 ceph-mon[67180]: pgmap v411: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:10.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:10 smithi122 ceph-mon[53211]: pgmap v412: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:11.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:10 smithi053 ceph-mon[67180]: pgmap v412: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:12.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:12 smithi122 ceph-mon[53211]: pgmap v413: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:12.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:12 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:58:13.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:12 smithi053 ceph-mon[67180]: pgmap v413: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:13.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:12 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:58:14.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:14 smithi122 ceph-mon[53211]: pgmap v414: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-02T04:58:15.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:14 smithi053 ceph-mon[67180]: pgmap v414: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-02T04:58:15.823 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:15 smithi053 ceph-mon[67180]: pgmap v415: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:15.825 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -e sha1=6cec793d9073f0594e2364b9b687ecaf22424cca -- bash -c 'ceph orch ps' 2024-04-02T04:58:16.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:15 smithi122 ceph-mon[53211]: pgmap v415: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:17.882 INFO:teuthology.orchestra.run.smithi053.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-02T04:58:17.882 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager.smithi053 smithi053 *:9093,9094 running (111s) 89s ago 21m 16.9M - 0.25.0 c8568f914cd2 15d2f21d8153 2024-04-02T04:58:17.882 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi053 smithi053 running (9m) 89s ago 21m 6660k - 19.0.0-1979-g6cec793d 78f830190839 4ce8915e0157 2024-04-02T04:58:17.882 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi122 smithi122 running (9m) 2m ago 21m 6660k - 19.0.0-1979-g6cec793d 78f830190839 e2b5369e6fdf 2024-04-02T04:58:17.883 INFO:teuthology.orchestra.run.smithi053.stdout:grafana.smithi053 smithi053 *:3000 running (95s) 89s ago 21m 61.0M - 9.4.12 2bacad6d85d8 6605ae1a36e4 2024-04-02T04:58:17.883 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi053.nfrfot smithi053 running (3m) 89s ago 16m 35.0M - 19.0.0-1979-g6cec793d 78f830190839 e7c0900d609e 2024-04-02T04:58:17.883 INFO:teuthology.orchestra.run.smithi053.stdout:mds.foofs.smithi122.mwoyoh smithi122 running (3m) 2m ago 15m 11.7M - 19.0.0-1979-g6cec793d 78f830190839 e0f9c7c343dd 2024-04-02T04:58:17.883 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi053.legdmv smithi053 *:8443,9283,8765 running (11m) 89s ago 23m 521M - 19.0.0-1979-g6cec793d 78f830190839 c12d516a31bc 2024-04-02T04:58:17.883 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi122.cpwlyk smithi122 *:8443,9283,8765 running (10m) 2m ago 21m 452M - 19.0.0-1979-g6cec793d 78f830190839 9a545c5e7d81 2024-04-02T04:58:17.883 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi053 smithi053 running (10m) 89s ago 23m 63.2M 2048M 19.0.0-1979-g6cec793d 78f830190839 f53a44f19c00 2024-04-02T04:58:17.883 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi122 smithi122 running (10m) 2m ago 21m 47.7M 2048M 19.0.0-1979-g6cec793d 78f830190839 44b200fed3c5 2024-04-02T04:58:17.883 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.0.0.smithi053.mhkdcb smithi053 *:2049 running (2m) 89s ago 15m 63.7M - 5.7 78f830190839 f15a31775645 2024-04-02T04:58:17.883 INFO:teuthology.orchestra.run.smithi053.stdout:nfs.foo.1.0.smithi122.mauvsb smithi122 *:2049 running (2m) 2m ago 15m 61.3M - 5.7 78f830190839 ca76a1d4d51f 2024-04-02T04:58:17.883 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi053 smithi053 *:9100 running (2m) 89s ago 21m 7856k - 1.5.0 0da6a335fe13 084c6c7e704a 2024-04-02T04:58:17.883 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi122 smithi122 *:9100 running (2m) 2m ago 20m 3418k - 1.5.0 0da6a335fe13 fd09a64a7e43 2024-04-02T04:58:17.883 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 smithi053 running (8m) 89s ago 20m 422M 1937M 19.0.0-1979-g6cec793d 78f830190839 490c346c7639 2024-04-02T04:58:17.883 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 smithi053 running (7m) 89s ago 19m 392M 1937M 19.0.0-1979-g6cec793d 78f830190839 e70e40949c2f 2024-04-02T04:58:17.884 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 smithi053 running (7m) 89s ago 19m 376M 1937M 19.0.0-1979-g6cec793d 78f830190839 4f053827e0dd 2024-04-02T04:58:17.884 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 smithi053 running (6m) 89s ago 19m 432M 1937M 19.0.0-1979-g6cec793d 78f830190839 5cd6de7a6535 2024-04-02T04:58:17.884 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 smithi122 running (5m) 2m ago 18m 367M 2704M 19.0.0-1979-g6cec793d 78f830190839 40d6ab1aac4b 2024-04-02T04:58:17.884 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 smithi122 running (5m) 2m ago 18m 333M 2704M 19.0.0-1979-g6cec793d 78f830190839 be26f425f51b 2024-04-02T04:58:17.884 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 smithi122 running (4m) 2m ago 17m 311M 2704M 19.0.0-1979-g6cec793d 78f830190839 aab18af876da 2024-04-02T04:58:17.884 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 smithi122 running (4m) 2m ago 17m 334M 2704M 19.0.0-1979-g6cec793d 78f830190839 86ee236f9426 2024-04-02T04:58:17.884 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus.smithi053 smithi053 *:9095 running (2m) 89s ago 20m 44.9M - 2.43.0 a07b618ecd1d 4add6ebe92ce 2024-04-02T04:58:18.546 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -e sha1=6cec793d9073f0594e2364b9b687ecaf22424cca -- bash -c 'ceph orch upgrade status' 2024-04-02T04:58:18.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:18 smithi053 ceph-mon[67180]: pgmap v416: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:18.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:18 smithi122 ceph-mon[53211]: pgmap v416: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:19.894 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:19 smithi053 ceph-mon[67180]: from='client.34588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:58:19.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:19 smithi122 ceph-mon[53211]: from='client.34588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:58:20.601 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:58:20.602 INFO:teuthology.orchestra.run.smithi053.stdout: "target_image": null, 2024-04-02T04:58:20.602 INFO:teuthology.orchestra.run.smithi053.stdout: "in_progress": false, 2024-04-02T04:58:20.602 INFO:teuthology.orchestra.run.smithi053.stdout: "which": "", 2024-04-02T04:58:20.602 INFO:teuthology.orchestra.run.smithi053.stdout: "services_complete": [], 2024-04-02T04:58:20.602 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": null, 2024-04-02T04:58:20.602 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "", 2024-04-02T04:58:20.602 INFO:teuthology.orchestra.run.smithi053.stdout: "is_paused": false 2024-04-02T04:58:20.602 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:58:20.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:20 smithi122 ceph-mon[53211]: pgmap v417: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:20.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:20 smithi122 ceph-mon[53211]: from='client.34592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:58:20.945 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:20 smithi053 ceph-mon[67180]: pgmap v417: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:20.945 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:20 smithi053 ceph-mon[67180]: from='client.34592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:58:21.299 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -e sha1=6cec793d9073f0594e2364b9b687ecaf22424cca -- bash -c 'ceph health detail' 2024-04-02T04:58:21.883 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:21 smithi053 ceph-mon[67180]: pgmap v418: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:21.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:21 smithi122 ceph-mon[53211]: pgmap v418: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:23.416 INFO:teuthology.orchestra.run.smithi053.stdout:HEALTH_OK 2024-04-02T04:58:23.744 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:23 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/2371897459' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:58:23.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:23 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/2371897459' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-02T04:58:24.000 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -e sha1=6cec793d9073f0594e2364b9b687ecaf22424cca -- bash -c 'ceph versions' 2024-04-02T04:58:24.655 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:24 smithi053 ceph-mon[67180]: pgmap v419: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-02T04:58:24.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:24 smithi122 ceph-mon[53211]: pgmap v419: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-02T04:58:26.058 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-04-02T04:58:26.058 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": { 2024-04-02T04:58:26.059 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:58:26.059 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:58:26.059 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": { 2024-04-02T04:58:26.059 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:58:26.059 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:58:26.059 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": { 2024-04-02T04:58:26.059 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 8 2024-04-02T04:58:26.059 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:58:26.059 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": { 2024-04-02T04:58:26.059 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 2 2024-04-02T04:58:26.059 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:58:26.059 INFO:teuthology.orchestra.run.smithi053.stdout: "overall": { 2024-04-02T04:58:26.059 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)": 14 2024-04-02T04:58:26.059 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-02T04:58:26.059 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-04-02T04:58:26.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:26 smithi053 ceph-mon[67180]: pgmap v420: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:26.808 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:26 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/3675996281' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:58:26.850 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -e sha1=6cec793d9073f0594e2364b9b687ecaf22424cca -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-04-02T04:58:26.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:26 smithi122 ceph-mon[53211]: pgmap v420: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:26.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:26 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/3675996281' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:58:27.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:27 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:58:27.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:27 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:58:28.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:28 smithi053 ceph-mon[67180]: pgmap v421: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:28.876 INFO:teuthology.orchestra.run.smithi053.stdout:true 2024-04-02T04:58:28.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:28 smithi122 ceph-mon[53211]: pgmap v421: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:29.480 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -e sha1=6cec793d9073f0594e2364b9b687ecaf22424cca -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-04-02T04:58:29.798 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:29 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/841956384' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:58:29.798 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:29 smithi053 ceph-mon[67180]: pgmap v422: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:29.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:29 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/841956384' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:58:29.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:29 smithi122 ceph-mon[53211]: pgmap v422: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:31.601 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph version 19.0.0-1979-g6cec793d (6cec793d9073f0594e2364b9b687ecaf22424cca) squid (dev)" 2024-04-02T04:58:31.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:31 smithi122 ceph-mon[53211]: pgmap v423: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:31.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:31 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/1868181980' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:58:31.921 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:31 smithi053 ceph-mon[67180]: pgmap v423: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:31.921 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:31 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/1868181980' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-02T04:58:32.235 INFO:teuthology.task.sequential:In sequential, running task cephadm.wait_for_service... 2024-04-02T04:58:32.236 INFO:tasks.cephadm:Waiting for ceph service nfs.foo to start (timeout 300)... 2024-04-02T04:58:32.236 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- ceph orch ls -f json 2024-04-02T04:58:34.174 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-02T04:58:34.174 INFO:teuthology.orchestra.run.smithi053.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-02T04:35:49.645669Z", "last_refresh": "2024-04-02T04:56:48.609568Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-02T04:35:44.306903Z", "last_refresh": "2024-04-02T04:55:53.629739Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "spec": {"anonymous_access": true, "protocol": "https"}, "status": {"created": "2024-04-02T04:35:47.354203Z", "last_refresh": "2024-04-02T04:56:48.609857Z", "ports": [3000], "running": 1, "size": 1}}, {"placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-02T04:42:12.446025Z", "last_refresh": "2024-04-02T04:55:53.630954Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-02T04:35:42.664861Z", "last_refresh": "2024-04-02T04:55:53.629920Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi053:172.21.15.53=smithi053", "smithi122:172.21.15.122=smithi122"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-02T04:36:30.245267Z", "last_refresh": "2024-04-02T04:55:53.630072Z", "running": 2, "size": 2}}, {"events": ["2024-04-02T04:57:55.115771Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 2049}, "status": {"created": "2024-04-02T04:42:32.501212Z", "last_refresh": "2024-04-02T04:55:53.631101Z", "ports": [2049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-02T04:35:48.479759Z", "last_refresh": "2024-04-02T04:55:53.630219Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808", "container_image_name": "quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae", "last_refresh": "2024-04-02T04:55:53.630367Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-02T04:35:45.496079Z", "last_refresh": "2024-04-02T04:56:48.610163Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-02T04:58:34.743 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:34 smithi053 ceph-mon[67180]: pgmap v424: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-02T04:58:34.770 INFO:tasks.cephadm:nfs.foo has 2/2 2024-04-02T04:58:34.771 DEBUG:teuthology.parallel:result is None 2024-04-02T04:58:34.771 INFO:teuthology.run_tasks:Running task vip.exec... 2024-04-02T04:58:34.781 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi053.front.sepia.ceph.com 2024-04-02T04:58:34.781 DEBUG:teuthology.orchestra.run.smithi053:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'umount /mnt/foo' 2024-04-02T04:58:34.810 INFO:teuthology.orchestra.run.smithi053.stderr:+ umount /mnt/foo 2024-04-02T04:58:34.822 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-02T04:58:34.832 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi053.front.sepia.ceph.com 2024-04-02T04:58:34.832 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- bash -c 'ceph nfs cluster ls | grep foo' 2024-04-02T04:58:34.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:34 smithi122 ceph-mon[53211]: pgmap v424: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-02T04:58:35.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:35 smithi053 ceph-mon[67180]: from='client.34612 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:58:35.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:35 smithi122 ceph-mon[53211]: from='client.34612 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T04:58:36.816 INFO:teuthology.orchestra.run.smithi053.stdout: "foo" 2024-04-02T04:58:36.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:36 smithi122 ceph-mon[53211]: pgmap v425: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:37.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:36 smithi053 ceph-mon[67180]: pgmap v425: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:37.568 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- bash -c 'ceph nfs export ls foo --detailed' 2024-04-02T04:58:37.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:37 smithi053 ceph-mon[67180]: from='client.34616 -' entity='client.admin' cmd=[{"prefix": "nfs cluster ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:58:37.807 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:37 smithi053 ceph-mon[67180]: pgmap v426: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:37.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:37 smithi122 ceph-mon[53211]: from='client.34616 -' entity='client.admin' cmd=[{"prefix": "nfs cluster ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:58:37.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:37 smithi122 ceph-mon[53211]: pgmap v426: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:39.618 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-04-02T04:58:39.618 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-04-02T04:58:39.618 INFO:teuthology.orchestra.run.smithi053.stdout: "access_type": "RW", 2024-04-02T04:58:39.618 INFO:teuthology.orchestra.run.smithi053.stdout: "clients": [], 2024-04-02T04:58:39.618 INFO:teuthology.orchestra.run.smithi053.stdout: "cluster_id": "foo", 2024-04-02T04:58:39.618 INFO:teuthology.orchestra.run.smithi053.stdout: "export_id": 1, 2024-04-02T04:58:39.619 INFO:teuthology.orchestra.run.smithi053.stdout: "fsal": { 2024-04-02T04:58:39.619 INFO:teuthology.orchestra.run.smithi053.stdout: "fs_name": "foofs", 2024-04-02T04:58:39.619 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "CEPH", 2024-04-02T04:58:39.619 INFO:teuthology.orchestra.run.smithi053.stdout: "user_id": "nfs.foo.1" 2024-04-02T04:58:39.619 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-04-02T04:58:39.619 INFO:teuthology.orchestra.run.smithi053.stdout: "path": "/", 2024-04-02T04:58:39.619 INFO:teuthology.orchestra.run.smithi053.stdout: "protocols": [ 2024-04-02T04:58:39.619 INFO:teuthology.orchestra.run.smithi053.stdout: 4 2024-04-02T04:58:39.619 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-04-02T04:58:39.619 INFO:teuthology.orchestra.run.smithi053.stdout: "pseudo": "/fake", 2024-04-02T04:58:39.619 INFO:teuthology.orchestra.run.smithi053.stdout: "security_label": true, 2024-04-02T04:58:39.619 INFO:teuthology.orchestra.run.smithi053.stdout: "squash": "none", 2024-04-02T04:58:39.619 INFO:teuthology.orchestra.run.smithi053.stdout: "transports": [ 2024-04-02T04:58:39.619 INFO:teuthology.orchestra.run.smithi053.stdout: "TCP" 2024-04-02T04:58:39.620 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-04-02T04:58:39.620 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-04-02T04:58:39.620 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-04-02T04:58:40.269 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- bash -c 'rados -p .nfs --all ls -' 2024-04-02T04:58:40.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:40 smithi122 ceph-mon[53211]: pgmap v427: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:40.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:40 smithi122 ceph-mon[53211]: from='client.34620 -' entity='client.admin' cmd=[{"prefix": "nfs export ls", "cluster_id": "foo", "detailed": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:58:40.948 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:40 smithi053 ceph-mon[67180]: pgmap v427: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:40.948 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:40 smithi053 ceph-mon[67180]: from='client.34620 -' entity='client.admin' cmd=[{"prefix": "nfs export ls", "cluster_id": "foo", "detailed": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T04:58:41.957 INFO:teuthology.orchestra.run.smithi053.stdout:foo rec-0000000000000004:nfs.foo.1 2024-04-02T04:58:41.957 INFO:teuthology.orchestra.run.smithi053.stdout:foo grace 2024-04-02T04:58:41.957 INFO:teuthology.orchestra.run.smithi053.stdout:foo rec-0000000000000004:nfs.foo.0 2024-04-02T04:58:41.957 INFO:teuthology.orchestra.run.smithi053.stdout:foo conf-nfs.foo 2024-04-02T04:58:41.957 INFO:teuthology.orchestra.run.smithi053.stdout:foo export-1 2024-04-02T04:58:42.684 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-02T04:58:42.694 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi053.front.sepia.ceph.com 2024-04-02T04:58:42.695 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 -- bash -c 'set -ex 2024-04-02T04:58:42.695 DEBUG:teuthology.orchestra.run.smithi053:> [[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]] 2024-04-02T04:58:42.695 DEBUG:teuthology.orchestra.run.smithi053:> ' 2024-04-02T04:58:42.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:42 smithi122 ceph-mon[53211]: pgmap v428: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:42.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:42 smithi122 ceph-mon[53211]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:58:42.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:42 smithi122 ceph-mon[53211]: mgrmap e42: smithi053.legdmv(active, since 10m), standbys: smithi122.cpwlyk 2024-04-02T04:58:43.008 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:42 smithi053 ceph-mon[67180]: pgmap v428: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-02T04:58:43.009 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:42 smithi053 ceph-mon[67180]: from='mgr.34104 172.21.15.53:0/3536280702' entity='mgr.smithi053.legdmv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-02T04:58:43.009 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:42 smithi053 ceph-mon[67180]: mgrmap e42: smithi053.legdmv(active, since 10m), standbys: smithi122.cpwlyk 2024-04-02T04:58:43.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:43 smithi122 ceph-mon[53211]: pgmap v429: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-02T04:58:43.992 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:43 smithi053 ceph-mon[67180]: pgmap v429: 97 pgs: 97 active+clean; 225 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-02T04:58:44.437 INFO:teuthology.orchestra.run.smithi053.stderr:++ ceph config get mgr mgr/cephadm/migration_current 2024-04-02T04:58:44.704 INFO:teuthology.orchestra.run.smithi053.stderr:+ [[ 6 -gt 2 ]] 2024-04-02T04:58:44.991 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:44 smithi053 ceph-mon[67180]: from='client.? 172.21.15.53:0/4222515613' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/cephadm/migration_current"}]: dispatch 2024-04-02T04:58:45.145 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:44 smithi122 ceph-mon[53211]: from='client.? 172.21.15.53:0/4222515613' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/cephadm/migration_current"}]: dispatch 2024-04-02T04:58:45.283 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-02T04:58:45.294 INFO:tasks.cephadm:Teardown begin 2024-04-02T04:58:45.294 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-02T04:58:45.322 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-02T04:58:45.359 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-02T04:58:45.359 DEBUG:teuthology.orchestra.run.smithi053:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-02T04:58:45.376 DEBUG:teuthology.orchestra.run.smithi122:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-02T04:58:45.417 INFO:tasks.cephadm:Stopping all daemons... 2024-04-02T04:58:45.417 INFO:tasks.cephadm.mon.smithi053:Stopping mon.smithi053... 2024-04-02T04:58:45.417 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@mon.smithi053 2024-04-02T04:58:45.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:45 smithi053 systemd[1]: Stopping Ceph mon.smithi053 for 48e16742-f0aa-11ee-b647-cb9ed24678a4... 2024-04-02T04:58:46.057 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:45 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi053[67176]: 2024-04-02T04:58:45.732+0000 7f5993959700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi053 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T04:58:46.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:45 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi053[67176]: 2024-04-02T04:58:45.732+0000 7f5993959700 -1 mon.smithi053@0(leader) e3 *** Got Signal Terminated *** 2024-04-02T04:58:46.058 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:45 smithi053 podman[111235]: 2024-04-02 04:58:45.966510463 +0000 UTC m=+0.364571994 container died f53a44f19c00fe0a4b3a7509bf212ae21125b103873d1bf43002f1ef2a420368 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi053, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-6cec793, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T04:58:46.557 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:46 smithi053 podman[111235]: 2024-04-02 04:58:46.366200174 +0000 UTC m=+0.764261694 container cleanup f53a44f19c00fe0a4b3a7509bf212ae21125b103873d1bf43002f1ef2a420368 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi053, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, RELEASE=squid-6cec793, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T04:58:46.558 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Apr 02 04:58:46 smithi053 bash[111235]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi053 2024-04-02T04:58:46.744 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@mon.smithi053.service' 2024-04-02T04:58:46.808 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T04:58:46.808 INFO:tasks.cephadm.mon.smithi053:Stopped mon.smithi053 2024-04-02T04:58:46.808 INFO:tasks.cephadm.mon.smithi122:Stopping mon.smithi122... 2024-04-02T04:58:46.808 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@mon.smithi122 2024-04-02T04:58:47.114 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:46 smithi122 systemd[1]: Stopping Ceph mon.smithi122 for 48e16742-f0aa-11ee-b647-cb9ed24678a4... 2024-04-02T04:58:47.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:47 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi122[53207]: 2024-04-02T04:58:47.109+0000 7f987ee13700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi122 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T04:58:47.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:47 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi122[53207]: 2024-04-02T04:58:47.109+0000 7f987ee13700 -1 mon.smithi122@1(peon) e3 *** Got Signal Terminated *** 2024-04-02T04:58:47.395 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:47 smithi122 podman[78504]: 2024-04-02 04:58:47.182015385 +0000 UTC m=+0.174245706 container died 44b200fed3c563ba8f5fbd9ed012b50611c7f0fa40150d8f77b52c5e4dbca5cf (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi122, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-6cec793, org.label-schema.build-date=20240326, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS) 2024-04-02T04:58:47.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:47 smithi122 podman[78504]: 2024-04-02 04:58:47.685697893 +0000 UTC m=+0.677928211 container cleanup 44b200fed3c563ba8f5fbd9ed012b50611c7f0fa40150d8f77b52c5e4dbca5cf (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi122, org.label-schema.license=GPLv2, RELEASE=squid-6cec793, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326) 2024-04-02T04:58:47.895 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 02 04:58:47 smithi122 bash[78504]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-mon-smithi122 2024-04-02T04:58:48.126 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@mon.smithi122.service' 2024-04-02T04:58:48.178 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T04:58:48.178 INFO:tasks.cephadm.mon.smithi122:Stopped mon.smithi122 2024-04-02T04:58:48.179 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-02T04:58:48.179 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.0 2024-04-02T04:58:48.503 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:58:48 smithi053 systemd[1]: Stopping Ceph osd.0 for 48e16742-f0aa-11ee-b647-cb9ed24678a4... 2024-04-02T04:58:48.807 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:58:48 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0[73096]: 2024-04-02T04:58:48.498+0000 7fc3ee7dd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T04:58:48.808 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:58:48 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0[73096]: 2024-04-02T04:58:48.498+0000 7fc3ee7dd700 -1 osd.0 152 *** Got signal Terminated *** 2024-04-02T04:58:48.808 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:58:48 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0[73096]: 2024-04-02T04:58:48.498+0000 7fc3ee7dd700 -1 osd.0 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T04:58:54.308 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:58:54 smithi053 podman[111368]: 2024-04-02 04:58:54.046706702 +0000 UTC m=+5.684984461 container died 490c346c76395bb103c58d16b72777fbb92f3be978fd3c8df6c017825fb4f9a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-6cec793, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-02T04:58:54.691 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:58:54 smithi053 podman[111368]: 2024-04-02 04:58:54.39511256 +0000 UTC m=+6.033390318 container cleanup 490c346c76395bb103c58d16b72777fbb92f3be978fd3c8df6c017825fb4f9a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0, maintainer=Guillaume Abrioux , RELEASE=squid-6cec793, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-02T04:58:54.692 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:58:54 smithi053 bash[111368]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0 2024-04-02T04:58:55.017 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:58:54 smithi053 podman[111418]: 2024-04-02 04:58:54.688166172 +0000 UTC m=+0.639129284 container remove 490c346c76395bb103c58d16b72777fbb92f3be978fd3c8df6c017825fb4f9a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-6cec793, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-02T04:58:55.308 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:58:55 smithi053 podman[111509]: 2024-04-02 04:58:54.916641129 +0000 UTC m=+0.018277245 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:58:55.308 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:58:55 smithi053 podman[111509]: 2024-04-02 04:58:55.030799616 +0000 UTC m=+0.132435704 container create 79cc6dd3206f752d4902023028415ff243b0db6a964bb9eeeb6bf18235eb1db3 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0-deactivate, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-6cec793, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-02T04:58:55.308 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:58:55 smithi053 podman[111509]: 2024-04-02 04:58:55.252984758 +0000 UTC m=+0.354620873 container init 79cc6dd3206f752d4902023028415ff243b0db6a964bb9eeeb6bf18235eb1db3 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0-deactivate, RELEASE=squid-6cec793, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-02T04:58:55.308 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:58:55 smithi053 podman[111509]: 2024-04-02 04:58:55.256566323 +0000 UTC m=+0.358202422 container start 79cc6dd3206f752d4902023028415ff243b0db6a964bb9eeeb6bf18235eb1db3 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-6cec793) 2024-04-02T04:58:55.807 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:58:55 smithi053 podman[111509]: 2024-04-02 04:58:55.335831389 +0000 UTC m=+0.437467493 container attach 79cc6dd3206f752d4902023028415ff243b0db6a964bb9eeeb6bf18235eb1db3 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-6cec793, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True) 2024-04-02T04:58:55.808 INFO:journalctl@ceph.osd.0.smithi053.stdout:Apr 02 04:58:55 smithi053 podman[111509]: 2024-04-02 04:58:55.556843547 +0000 UTC m=+0.658479648 container died 79cc6dd3206f752d4902023028415ff243b0db6a964bb9eeeb6bf18235eb1db3 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-6cec793, ceph=True, org.label-schema.build-date=20240326, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-02T04:58:56.257 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.0.service' 2024-04-02T04:58:56.297 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T04:58:56.297 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-02T04:58:56.297 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-02T04:58:56.297 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.1 2024-04-02T04:58:56.557 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:58:56 smithi053 systemd[1]: Stopping Ceph osd.1 for 48e16742-f0aa-11ee-b647-cb9ed24678a4... 2024-04-02T04:58:57.057 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:58:56 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1[75630]: 2024-04-02T04:58:56.667+0000 7f539e46e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T04:58:57.058 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:58:56 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1[75630]: 2024-04-02T04:58:56.667+0000 7f539e46e700 -1 osd.1 152 *** Got signal Terminated *** 2024-04-02T04:58:57.058 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:58:56 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1[75630]: 2024-04-02T04:58:56.667+0000 7f539e46e700 -1 osd.1 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T04:59:02.308 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:59:02 smithi053 podman[111613]: 2024-04-02 04:59:02.051346074 +0000 UTC m=+5.518861362 container died e70e40949c2f34a2199cd5db9474ccfd4290ae038d7f3d781e86d139271fc92b (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1, org.label-schema.build-date=20240326, RELEASE=squid-6cec793, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T04:59:02.664 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:59:02 smithi053 podman[111613]: 2024-04-02 04:59:02.426215487 +0000 UTC m=+5.893730762 container cleanup e70e40949c2f34a2199cd5db9474ccfd4290ae038d7f3d781e86d139271fc92b (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-6cec793, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T04:59:02.664 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:59:02 smithi053 bash[111613]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1 2024-04-02T04:59:03.000 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:59:02 smithi053 podman[111631]: 2024-04-02 04:59:02.660829372 +0000 UTC m=+0.613615147 container remove e70e40949c2f34a2199cd5db9474ccfd4290ae038d7f3d781e86d139271fc92b (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1, RELEASE=squid-6cec793, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326) 2024-04-02T04:59:03.259 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:59:02 smithi053 podman[111721]: 2024-04-02 04:59:02.900280022 +0000 UTC m=+0.018555372 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:59:03.259 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:59:03 smithi053 podman[111721]: 2024-04-02 04:59:03.045091858 +0000 UTC m=+0.163367214 container create d557396972a11de5cc2fe0fab8fe67679e605fff4f3a24aad31cf96e971108cb (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-6cec793, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326) 2024-04-02T04:59:03.259 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:59:03 smithi053 podman[111721]: 2024-04-02 04:59:03.25576533 +0000 UTC m=+0.374040680 container init d557396972a11de5cc2fe0fab8fe67679e605fff4f3a24aad31cf96e971108cb (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, RELEASE=squid-6cec793) 2024-04-02T04:59:03.558 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:59:03 smithi053 podman[111721]: 2024-04-02 04:59:03.260035576 +0000 UTC m=+0.378310934 container start d557396972a11de5cc2fe0fab8fe67679e605fff4f3a24aad31cf96e971108cb (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-6cec793, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326) 2024-04-02T04:59:03.558 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:59:03 smithi053 podman[111721]: 2024-04-02 04:59:03.347543998 +0000 UTC m=+0.465819366 container attach d557396972a11de5cc2fe0fab8fe67679e605fff4f3a24aad31cf96e971108cb (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-6cec793, org.label-schema.build-date=20240326, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-02T04:59:03.913 INFO:journalctl@ceph.osd.1.smithi053.stdout:Apr 02 04:59:03 smithi053 podman[111721]: 2024-04-02 04:59:03.571080902 +0000 UTC m=+0.689356257 container died d557396972a11de5cc2fe0fab8fe67679e605fff4f3a24aad31cf96e971108cb (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-6cec793) 2024-04-02T04:59:04.220 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.1.service' 2024-04-02T04:59:04.260 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T04:59:04.261 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-02T04:59:04.261 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-02T04:59:04.261 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.2 2024-04-02T04:59:04.557 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:59:04 smithi053 systemd[1]: Stopping Ceph osd.2 for 48e16742-f0aa-11ee-b647-cb9ed24678a4... 2024-04-02T04:59:05.057 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:59:04 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2[78304]: 2024-04-02T04:59:04.646+0000 7fc2ff3aa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T04:59:05.058 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:59:04 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2[78304]: 2024-04-02T04:59:04.646+0000 7fc2ff3aa700 -1 osd.2 152 *** Got signal Terminated *** 2024-04-02T04:59:05.058 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:59:04 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2[78304]: 2024-04-02T04:59:04.646+0000 7fc2ff3aa700 -1 osd.2 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T04:59:10.451 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:59:10 smithi053 podman[111826]: 2024-04-02 04:59:10.1682193 +0000 UTC m=+5.661569163 container died 4f053827e0dd38ab53819701163449278eb07e84ece8c26b1ce8292bd7b7f98f (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, RELEASE=squid-6cec793, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-02T04:59:10.808 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:59:10 smithi053 podman[111826]: 2024-04-02 04:59:10.50371231 +0000 UTC m=+5.997062181 container cleanup 4f053827e0dd38ab53819701163449278eb07e84ece8c26b1ce8292bd7b7f98f (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, RELEASE=squid-6cec793, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True) 2024-04-02T04:59:10.808 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:59:10 smithi053 bash[111826]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2 2024-04-02T04:59:10.808 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:59:10 smithi053 podman[111844]: 2024-04-02 04:59:10.671621038 +0000 UTC m=+0.499988823 container remove 4f053827e0dd38ab53819701163449278eb07e84ece8c26b1ce8292bd7b7f98f (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, RELEASE=squid-6cec793, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-02T04:59:11.416 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:59:11 smithi053 podman[111971]: 2024-04-02 04:59:11.062430213 +0000 UTC m=+0.018099709 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:59:11.416 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:59:11 smithi053 podman[111971]: 2024-04-02 04:59:11.189299869 +0000 UTC m=+0.144969355 container create 3982eba2056dc467460da4a5af46eaf4394863d950bdf40ba610351efc60b9a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, RELEASE=squid-6cec793, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T04:59:11.416 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:59:11 smithi053 podman[111971]: 2024-04-02 04:59:11.408436448 +0000 UTC m=+0.364105947 container init 3982eba2056dc467460da4a5af46eaf4394863d950bdf40ba610351efc60b9a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2-deactivate, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-6cec793, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=) 2024-04-02T04:59:11.416 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:59:11 smithi053 podman[111971]: 2024-04-02 04:59:11.412190519 +0000 UTC m=+0.367860008 container start 3982eba2056dc467460da4a5af46eaf4394863d950bdf40ba610351efc60b9a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2-deactivate, io.buildah.version=1.33.5, RELEASE=squid-6cec793, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326) 2024-04-02T04:59:11.758 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:59:11 smithi053 podman[111971]: 2024-04-02 04:59:11.525528972 +0000 UTC m=+0.481198470 container attach 3982eba2056dc467460da4a5af46eaf4394863d950bdf40ba610351efc60b9a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-6cec793, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, io.buildah.version=1.33.5) 2024-04-02T04:59:12.057 INFO:journalctl@ceph.osd.2.smithi053.stdout:Apr 02 04:59:11 smithi053 podman[111971]: 2024-04-02 04:59:11.754820031 +0000 UTC m=+0.710489514 container died 3982eba2056dc467460da4a5af46eaf4394863d950bdf40ba610351efc60b9a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-2-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-6cec793, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS) 2024-04-02T04:59:12.418 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.2.service' 2024-04-02T04:59:12.458 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T04:59:12.458 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-02T04:59:12.458 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-02T04:59:12.458 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.3 2024-04-02T04:59:12.807 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:59:12 smithi053 systemd[1]: Stopping Ceph osd.3 for 48e16742-f0aa-11ee-b647-cb9ed24678a4... 2024-04-02T04:59:13.307 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:59:12 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3[80843]: 2024-04-02T04:59:12.806+0000 7f0e3b202700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T04:59:13.308 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:59:12 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3[80843]: 2024-04-02T04:59:12.806+0000 7f0e3b202700 -1 osd.3 152 *** Got signal Terminated *** 2024-04-02T04:59:13.308 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:59:12 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3[80843]: 2024-04-02T04:59:12.806+0000 7f0e3b202700 -1 osd.3 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T04:59:15.127 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:59:14 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4[58092]: 2024-04-02T04:59:14.738+0000 7f3687212700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:53.428488+0000 front 2024-04-02T04:58:53.428525+0000 (oldest deadline 2024-04-02T04:59:14.528745+0000) 2024-04-02T04:59:15.395 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:15 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:15.123+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.270997+0000 front 2024-04-02T04:58:50.271075+0000 (oldest deadline 2024-04-02T04:59:14.371167+0000) 2024-04-02T04:59:16.145 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:59:15 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4[58092]: 2024-04-02T04:59:15.720+0000 7f3687212700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:53.428488+0000 front 2024-04-02T04:58:53.428525+0000 (oldest deadline 2024-04-02T04:59:14.528745+0000) 2024-04-02T04:59:16.645 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:16 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:59:16.232+0000 7f45bf786700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:52.720639+0000 front 2024-04-02T04:58:52.720588+0000 (oldest deadline 2024-04-02T04:59:16.220734+0000) 2024-04-02T04:59:16.646 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:16 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:16.161+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.270997+0000 front 2024-04-02T04:58:50.271075+0000 (oldest deadline 2024-04-02T04:59:14.371167+0000) 2024-04-02T04:59:17.145 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:59:16 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4[58092]: 2024-04-02T04:59:16.748+0000 7f3687212700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:53.428488+0000 front 2024-04-02T04:58:53.428525+0000 (oldest deadline 2024-04-02T04:59:14.528745+0000) 2024-04-02T04:59:17.581 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:17 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:59:17.233+0000 7f45bf786700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:52.720639+0000 front 2024-04-02T04:58:52.720588+0000 (oldest deadline 2024-04-02T04:59:16.220734+0000) 2024-04-02T04:59:17.581 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:17 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:17.184+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.270997+0000 front 2024-04-02T04:58:50.271075+0000 (oldest deadline 2024-04-02T04:59:14.371167+0000) 2024-04-02T04:59:17.895 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:59:17 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4[58092]: 2024-04-02T04:59:17.708+0000 7f3687212700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:53.428488+0000 front 2024-04-02T04:58:53.428525+0000 (oldest deadline 2024-04-02T04:59:14.528745+0000) 2024-04-02T04:59:17.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:17 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:17.577+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.967123+0000 front 2024-04-02T04:58:50.967101+0000 (oldest deadline 2024-04-02T04:59:16.867341+0000) 2024-04-02T04:59:18.057 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:59:17 smithi053 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3[80843]: 2024-04-02T04:59:17.801+0000 7f0e337ee700 -1 osd.3 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:53.141865+0000 front 2024-04-02T04:58:53.141951+0000 (oldest deadline 2024-04-02T04:59:17.241991+0000) 2024-04-02T04:59:18.585 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:18 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:59:18.207+0000 7f45bf786700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:52.720639+0000 front 2024-04-02T04:58:52.720588+0000 (oldest deadline 2024-04-02T04:59:16.220734+0000) 2024-04-02T04:59:18.585 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:18 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:18.221+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.270997+0000 front 2024-04-02T04:58:50.271075+0000 (oldest deadline 2024-04-02T04:59:14.371167+0000) 2024-04-02T04:59:18.619 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:59:18 smithi053 podman[112075]: 2024-04-02 04:59:18.33736653 +0000 UTC m=+5.635983391 container died 5cd6de7a65352d2b97c16dbd45c24243e03b570d833f9889302776e0df666e30 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, RELEASE=squid-6cec793, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-02T04:59:18.895 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:59:18 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4[58092]: 2024-04-02T04:59:18.679+0000 7f3687212700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:53.428488+0000 front 2024-04-02T04:58:53.428525+0000 (oldest deadline 2024-04-02T04:59:14.528745+0000) 2024-04-02T04:59:18.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:18 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:18.580+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.967123+0000 front 2024-04-02T04:58:50.967101+0000 (oldest deadline 2024-04-02T04:59:16.867341+0000) 2024-04-02T04:59:18.961 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:59:18 smithi053 podman[112075]: 2024-04-02 04:59:18.71470546 +0000 UTC m=+6.013322305 container cleanup 5cd6de7a65352d2b97c16dbd45c24243e03b570d833f9889302776e0df666e30 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-6cec793, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T04:59:18.961 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:59:18 smithi053 bash[112075]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3 2024-04-02T04:59:19.295 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:59:18 smithi053 podman[112093]: 2024-04-02 04:59:18.957611945 +0000 UTC m=+0.615125711 container remove 5cd6de7a65352d2b97c16dbd45c24243e03b570d833f9889302776e0df666e30 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, RELEASE=squid-6cec793, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-02T04:59:19.558 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:59:19 smithi053 podman[112186]: 2024-04-02 04:59:19.2920296 +0000 UTC m=+0.115577807 container create 72351abfa302f4c5311c277d9980223f0c2bbc01bb7d652c6feaab922faa4659 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-6cec793, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-02T04:59:19.558 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:59:19 smithi053 podman[112186]: 2024-04-02 04:59:19.195850059 +0000 UTC m=+0.019398281 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:59:19.558 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:59:19 smithi053 podman[112186]: 2024-04-02 04:59:19.527686847 +0000 UTC m=+0.351235065 container init 72351abfa302f4c5311c277d9980223f0c2bbc01bb7d652c6feaab922faa4659 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-6cec793) 2024-04-02T04:59:19.558 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:59:19 smithi053 podman[112186]: 2024-04-02 04:59:19.532001365 +0000 UTC m=+0.355549571 container start 72351abfa302f4c5311c277d9980223f0c2bbc01bb7d652c6feaab922faa4659 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-6cec793, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-02T04:59:19.604 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:19 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:59:19.214+0000 7f45bf786700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:52.720639+0000 front 2024-04-02T04:58:52.720588+0000 (oldest deadline 2024-04-02T04:59:16.220734+0000) 2024-04-02T04:59:19.604 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:19 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:19.261+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.270997+0000 front 2024-04-02T04:58:50.271075+0000 (oldest deadline 2024-04-02T04:59:14.371167+0000) 2024-04-02T04:59:19.895 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:59:19 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4[58092]: 2024-04-02T04:59:19.658+0000 7f3687212700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:53.428488+0000 front 2024-04-02T04:58:53.428525+0000 (oldest deadline 2024-04-02T04:59:14.528745+0000) 2024-04-02T04:59:19.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:19 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:19.607+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.967123+0000 front 2024-04-02T04:58:50.967101+0000 (oldest deadline 2024-04-02T04:59:16.867341+0000) 2024-04-02T04:59:20.057 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:59:19 smithi053 podman[112186]: 2024-04-02 04:59:19.619600794 +0000 UTC m=+0.443149004 container attach 72351abfa302f4c5311c277d9980223f0c2bbc01bb7d652c6feaab922faa4659 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, RELEASE=squid-6cec793, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD) 2024-04-02T04:59:20.058 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:59:19 smithi053 podman[112186]: 2024-04-02 04:59:19.841710773 +0000 UTC m=+0.665258983 container died 72351abfa302f4c5311c277d9980223f0c2bbc01bb7d652c6feaab922faa4659 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, RELEASE=squid-6cec793, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-02T04:59:20.268 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:20 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:59:20.213+0000 7f45bf786700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:52.720639+0000 front 2024-04-02T04:58:52.720588+0000 (oldest deadline 2024-04-02T04:59:16.220734+0000) 2024-04-02T04:59:20.389 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.3.service' 2024-04-02T04:59:20.414 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:59:20 smithi053 podman[112211]: 2024-04-02 04:59:20.364654299 +0000 UTC m=+0.515033171 container remove 72351abfa302f4c5311c277d9980223f0c2bbc01bb7d652c6feaab922faa4659 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-3-deactivate, RELEASE=squid-6cec793, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_BRANCH=HEAD) 2024-04-02T04:59:20.414 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:59:20 smithi053 systemd[1]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.3.service: Deactivated successfully. 2024-04-02T04:59:20.414 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:59:20 smithi053 systemd[1]: Stopped Ceph osd.3 for 48e16742-f0aa-11ee-b647-cb9ed24678a4. 2024-04-02T04:59:20.414 INFO:journalctl@ceph.osd.3.smithi053.stdout:Apr 02 04:59:20 smithi053 systemd[1]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.3.service: Consumed 1min 45.306s CPU time. 2024-04-02T04:59:20.430 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T04:59:20.430 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-02T04:59:20.430 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-02T04:59:20.431 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.4 2024-04-02T04:59:20.597 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:59:20 smithi122 systemd[1]: Stopping Ceph osd.4 for 48e16742-f0aa-11ee-b647-cb9ed24678a4... 2024-04-02T04:59:20.597 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:20 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:20.264+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.270997+0000 front 2024-04-02T04:58:50.271075+0000 (oldest deadline 2024-04-02T04:59:14.371167+0000) 2024-04-02T04:59:20.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:20 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:20.592+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.967123+0000 front 2024-04-02T04:58:50.967101+0000 (oldest deadline 2024-04-02T04:59:16.867341+0000) 2024-04-02T04:59:20.896 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:59:20 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4[58092]: 2024-04-02T04:59:20.707+0000 7f3687212700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:53.428488+0000 front 2024-04-02T04:58:53.428525+0000 (oldest deadline 2024-04-02T04:59:14.528745+0000) 2024-04-02T04:59:20.896 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:59:20 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4[58092]: 2024-04-02T04:59:20.737+0000 7f368ec26700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T04:59:20.896 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:59:20 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4[58092]: 2024-04-02T04:59:20.737+0000 7f368ec26700 -1 osd.4 152 *** Got signal Terminated *** 2024-04-02T04:59:20.896 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:59:20 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4[58092]: 2024-04-02T04:59:20.737+0000 7f368ec26700 -1 osd.4 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T04:59:21.608 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:21 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:59:21.234+0000 7f45bf786700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:52.720639+0000 front 2024-04-02T04:58:52.720588+0000 (oldest deadline 2024-04-02T04:59:16.220734+0000) 2024-04-02T04:59:21.609 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:21 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:21.312+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.270997+0000 front 2024-04-02T04:58:50.271075+0000 (oldest deadline 2024-04-02T04:59:14.371167+0000) 2024-04-02T04:59:21.895 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:59:21 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4[58092]: 2024-04-02T04:59:21.753+0000 7f3687212700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:53.428488+0000 front 2024-04-02T04:58:53.428525+0000 (oldest deadline 2024-04-02T04:59:14.528745+0000) 2024-04-02T04:59:21.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:21 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:21.604+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.967123+0000 front 2024-04-02T04:58:50.967101+0000 (oldest deadline 2024-04-02T04:59:16.867341+0000) 2024-04-02T04:59:22.574 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:22 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:59:22.276+0000 7f45bf786700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:52.720639+0000 front 2024-04-02T04:58:52.720588+0000 (oldest deadline 2024-04-02T04:59:16.220734+0000) 2024-04-02T04:59:22.574 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:22 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:22.283+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.270997+0000 front 2024-04-02T04:58:50.271075+0000 (oldest deadline 2024-04-02T04:59:14.371167+0000) 2024-04-02T04:59:22.895 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:59:22 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4[58092]: 2024-04-02T04:59:22.741+0000 7f3687212700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:53.428488+0000 front 2024-04-02T04:58:53.428525+0000 (oldest deadline 2024-04-02T04:59:14.528745+0000) 2024-04-02T04:59:22.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:22 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:22.569+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.967123+0000 front 2024-04-02T04:58:50.967101+0000 (oldest deadline 2024-04-02T04:59:16.867341+0000) 2024-04-02T04:59:22.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:22 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:22.569+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:56.867795+0000 front 2024-04-02T04:58:56.867864+0000 (oldest deadline 2024-04-02T04:59:22.168002+0000) 2024-04-02T04:59:23.569 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:23 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:23.302+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.270997+0000 front 2024-04-02T04:58:50.271075+0000 (oldest deadline 2024-04-02T04:59:14.371167+0000) 2024-04-02T04:59:23.569 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:23 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:59:23.286+0000 7f45bf786700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:52.720639+0000 front 2024-04-02T04:58:52.720588+0000 (oldest deadline 2024-04-02T04:59:16.220734+0000) 2024-04-02T04:59:23.569 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:23 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:59:23.286+0000 7f45bf786700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:59.721447+0000 front 2024-04-02T04:58:59.721523+0000 (oldest deadline 2024-04-02T04:59:22.621507+0000) 2024-04-02T04:59:23.895 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:59:23 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4[58092]: 2024-04-02T04:59:23.732+0000 7f3687212700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:53.428488+0000 front 2024-04-02T04:58:53.428525+0000 (oldest deadline 2024-04-02T04:59:14.528745+0000) 2024-04-02T04:59:23.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:23 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:23.565+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.967123+0000 front 2024-04-02T04:58:50.967101+0000 (oldest deadline 2024-04-02T04:59:16.867341+0000) 2024-04-02T04:59:23.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:23 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:23.565+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:56.867795+0000 front 2024-04-02T04:58:56.867864+0000 (oldest deadline 2024-04-02T04:59:22.168002+0000) 2024-04-02T04:59:24.584 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:24 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:24.325+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.270997+0000 front 2024-04-02T04:58:50.271075+0000 (oldest deadline 2024-04-02T04:59:14.371167+0000) 2024-04-02T04:59:24.585 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:24 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:59:24.266+0000 7f45bf786700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:52.720639+0000 front 2024-04-02T04:58:52.720588+0000 (oldest deadline 2024-04-02T04:59:16.220734+0000) 2024-04-02T04:59:24.585 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:24 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:59:24.266+0000 7f45bf786700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:59.721447+0000 front 2024-04-02T04:58:59.721523+0000 (oldest deadline 2024-04-02T04:59:22.621507+0000) 2024-04-02T04:59:24.895 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:59:24 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4[58092]: 2024-04-02T04:59:24.749+0000 7f3687212700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:53.428488+0000 front 2024-04-02T04:58:53.428525+0000 (oldest deadline 2024-04-02T04:59:14.528745+0000) 2024-04-02T04:59:24.895 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:59:24 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4[58092]: 2024-04-02T04:59:24.749+0000 7f3687212700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:59.229583+0000 front 2024-04-02T04:58:59.229616+0000 (oldest deadline 2024-04-02T04:59:24.529713+0000) 2024-04-02T04:59:24.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:24 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:24.580+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.967123+0000 front 2024-04-02T04:58:50.967101+0000 (oldest deadline 2024-04-02T04:59:16.867341+0000) 2024-04-02T04:59:24.896 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:24 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:24.580+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:56.867795+0000 front 2024-04-02T04:58:56.867864+0000 (oldest deadline 2024-04-02T04:59:22.168002+0000) 2024-04-02T04:59:25.566 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:25 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:59:25.286+0000 7f45bf786700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:52.720639+0000 front 2024-04-02T04:58:52.720588+0000 (oldest deadline 2024-04-02T04:59:16.220734+0000) 2024-04-02T04:59:25.566 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:25 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:59:25.286+0000 7f45bf786700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:59.721447+0000 front 2024-04-02T04:58:59.721523+0000 (oldest deadline 2024-04-02T04:59:22.621507+0000) 2024-04-02T04:59:25.566 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:25 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:25.359+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.270997+0000 front 2024-04-02T04:58:50.271075+0000 (oldest deadline 2024-04-02T04:59:14.371167+0000) 2024-04-02T04:59:25.566 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:25 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:25.359+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:59:00.172643+0000 front 2024-04-02T04:59:00.172743+0000 (oldest deadline 2024-04-02T04:59:24.872692+0000) 2024-04-02T04:59:25.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:25 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:25.562+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.967123+0000 front 2024-04-02T04:58:50.967101+0000 (oldest deadline 2024-04-02T04:59:16.867341+0000) 2024-04-02T04:59:25.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:25 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:25.562+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:56.867795+0000 front 2024-04-02T04:58:56.867864+0000 (oldest deadline 2024-04-02T04:59:22.168002+0000) 2024-04-02T04:59:26.395 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:59:26 smithi122 podman[78701]: 2024-04-02 04:59:26.138517822 +0000 UTC m=+5.512604259 container died 40d6ab1aac4b114be6c69ce2ec01ed0e4b5ad9d84c2f66cbb482ec407c213a01 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-6cec793, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2) 2024-04-02T04:59:26.395 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:26 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:59:26.244+0000 7f45bf786700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:52.720639+0000 front 2024-04-02T04:58:52.720588+0000 (oldest deadline 2024-04-02T04:59:16.220734+0000) 2024-04-02T04:59:26.395 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:26 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:59:26.244+0000 7f45bf786700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:59.721447+0000 front 2024-04-02T04:58:59.721523+0000 (oldest deadline 2024-04-02T04:59:22.621507+0000) 2024-04-02T04:59:26.396 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:26 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:26.387+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.270997+0000 front 2024-04-02T04:58:50.271075+0000 (oldest deadline 2024-04-02T04:59:14.371167+0000) 2024-04-02T04:59:26.396 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:26 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:26.387+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:59:00.172643+0000 front 2024-04-02T04:59:00.172743+0000 (oldest deadline 2024-04-02T04:59:24.872692+0000) 2024-04-02T04:59:26.776 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:59:26 smithi122 podman[78701]: 2024-04-02 04:59:26.571790373 +0000 UTC m=+5.945876801 container cleanup 40d6ab1aac4b114be6c69ce2ec01ed0e4b5ad9d84c2f66cbb482ec407c213a01 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240326, RELEASE=squid-6cec793, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T04:59:26.776 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:59:26 smithi122 bash[78701]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4 2024-04-02T04:59:26.777 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:26 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:26.559+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.967123+0000 front 2024-04-02T04:58:50.967101+0000 (oldest deadline 2024-04-02T04:59:16.867341+0000) 2024-04-02T04:59:26.777 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:26 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:26.559+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:56.867795+0000 front 2024-04-02T04:58:56.867864+0000 (oldest deadline 2024-04-02T04:59:22.168002+0000) 2024-04-02T04:59:27.146 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:59:26 smithi122 podman[78721]: 2024-04-02 04:59:26.773169971 +0000 UTC m=+0.629097034 container remove 40d6ab1aac4b114be6c69ce2ec01ed0e4b5ad9d84c2f66cbb482ec407c213a01 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4, RELEASE=squid-6cec793, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T04:59:27.479 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:27 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:59:27.234+0000 7f45bf786700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:52.720639+0000 front 2024-04-02T04:58:52.720588+0000 (oldest deadline 2024-04-02T04:59:16.220734+0000) 2024-04-02T04:59:27.479 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:27 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:59:27.234+0000 7f45bf786700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:59.721447+0000 front 2024-04-02T04:58:59.721523+0000 (oldest deadline 2024-04-02T04:59:22.621507+0000) 2024-04-02T04:59:27.480 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:27 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:27.406+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.270997+0000 front 2024-04-02T04:58:50.271075+0000 (oldest deadline 2024-04-02T04:59:14.371167+0000) 2024-04-02T04:59:27.480 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:27 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:27.406+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:59:00.172643+0000 front 2024-04-02T04:59:00.172743+0000 (oldest deadline 2024-04-02T04:59:24.872692+0000) 2024-04-02T04:59:27.480 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:59:27 smithi122 podman[78811]: 2024-04-02 04:59:27.081464765 +0000 UTC m=+0.018057567 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:59:27.480 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:59:27 smithi122 podman[78811]: 2024-04-02 04:59:27.190848869 +0000 UTC m=+0.127441700 container create 8beaba3d18db9654d4122e05b550d6372ae2a40df45c713514e1a0ee6a2bece0 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-6cec793, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T04:59:27.480 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:59:27 smithi122 podman[78811]: 2024-04-02 04:59:27.385257003 +0000 UTC m=+0.321849792 container init 8beaba3d18db9654d4122e05b550d6372ae2a40df45c713514e1a0ee6a2bece0 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-6cec793, CEPH_POINT_RELEASE=) 2024-04-02T04:59:27.480 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:59:27 smithi122 podman[78811]: 2024-04-02 04:59:27.388627236 +0000 UTC m=+0.325220024 container start 8beaba3d18db9654d4122e05b550d6372ae2a40df45c713514e1a0ee6a2bece0 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240326, RELEASE=squid-6cec793, io.buildah.version=1.33.5) 2024-04-02T04:59:27.480 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:59:27 smithi122 podman[78811]: 2024-04-02 04:59:27.476514871 +0000 UTC m=+0.413107675 container attach 8beaba3d18db9654d4122e05b550d6372ae2a40df45c713514e1a0ee6a2bece0 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-6cec793, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2) 2024-04-02T04:59:27.895 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:59:27 smithi122 podman[78811]: 2024-04-02 04:59:27.686469099 +0000 UTC m=+0.623061890 container died 8beaba3d18db9654d4122e05b550d6372ae2a40df45c713514e1a0ee6a2bece0 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4-deactivate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, RELEASE=squid-6cec793, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-02T04:59:27.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:27 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:27.607+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.967123+0000 front 2024-04-02T04:58:50.967101+0000 (oldest deadline 2024-04-02T04:59:16.867341+0000) 2024-04-02T04:59:27.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:27 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:27.607+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:56.867795+0000 front 2024-04-02T04:58:56.867864+0000 (oldest deadline 2024-04-02T04:59:22.168002+0000) 2024-04-02T04:59:28.264 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 02 04:59:28 smithi122 podman[78836]: 2024-04-02 04:59:28.238458632 +0000 UTC m=+0.541844572 container remove 8beaba3d18db9654d4122e05b550d6372ae2a40df45c713514e1a0ee6a2bece0 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, RELEASE=squid-6cec793, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-02T04:59:28.264 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:28 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:59:28.211+0000 7f45bf786700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:52.720639+0000 front 2024-04-02T04:58:52.720588+0000 (oldest deadline 2024-04-02T04:59:16.220734+0000) 2024-04-02T04:59:28.264 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:28 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:59:28.211+0000 7f45bf786700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:59.721447+0000 front 2024-04-02T04:58:59.721523+0000 (oldest deadline 2024-04-02T04:59:22.621507+0000) 2024-04-02T04:59:28.269 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.4.service' 2024-04-02T04:59:28.311 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T04:59:28.312 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-02T04:59:28.312 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-02T04:59:28.312 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.5 2024-04-02T04:59:28.606 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:28 smithi122 systemd[1]: Stopping Ceph osd.5 for 48e16742-f0aa-11ee-b647-cb9ed24678a4... 2024-04-02T04:59:28.607 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:28 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:28.406+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.270997+0000 front 2024-04-02T04:58:50.271075+0000 (oldest deadline 2024-04-02T04:59:14.371167+0000) 2024-04-02T04:59:28.607 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:28 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:28.406+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:59:00.172643+0000 front 2024-04-02T04:59:00.172743+0000 (oldest deadline 2024-04-02T04:59:24.872692+0000) 2024-04-02T04:59:28.607 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:28 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:28.603+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.967123+0000 front 2024-04-02T04:58:50.967101+0000 (oldest deadline 2024-04-02T04:59:16.867341+0000) 2024-04-02T04:59:28.607 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:28 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:28.603+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:56.867795+0000 front 2024-04-02T04:58:56.867864+0000 (oldest deadline 2024-04-02T04:59:22.168002+0000) 2024-04-02T04:59:28.895 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:28 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:59:28.636+0000 7f45c3192700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T04:59:28.895 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:28 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:59:28.636+0000 7f45c3192700 -1 osd.5 152 *** Got signal Terminated *** 2024-04-02T04:59:28.895 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:28 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:59:28.636+0000 7f45c3192700 -1 osd.5 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T04:59:29.576 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:29 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:59:29.257+0000 7f45bf786700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:52.720639+0000 front 2024-04-02T04:58:52.720588+0000 (oldest deadline 2024-04-02T04:59:16.220734+0000) 2024-04-02T04:59:29.576 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:29 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:59:29.257+0000 7f45bf786700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:59.721447+0000 front 2024-04-02T04:58:59.721523+0000 (oldest deadline 2024-04-02T04:59:22.621507+0000) 2024-04-02T04:59:29.577 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:29 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:29.363+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.270997+0000 front 2024-04-02T04:58:50.271075+0000 (oldest deadline 2024-04-02T04:59:14.371167+0000) 2024-04-02T04:59:29.577 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:29 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:29.363+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:59:00.172643+0000 front 2024-04-02T04:59:00.172743+0000 (oldest deadline 2024-04-02T04:59:24.872692+0000) 2024-04-02T04:59:29.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:29 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:29.572+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.967123+0000 front 2024-04-02T04:58:50.967101+0000 (oldest deadline 2024-04-02T04:59:16.867341+0000) 2024-04-02T04:59:29.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:29 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:29.572+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:56.867795+0000 front 2024-04-02T04:58:56.867864+0000 (oldest deadline 2024-04-02T04:59:22.168002+0000) 2024-04-02T04:59:30.543 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:30 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:59:30.305+0000 7f45bf786700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:52.720639+0000 front 2024-04-02T04:58:52.720588+0000 (oldest deadline 2024-04-02T04:59:16.220734+0000) 2024-04-02T04:59:30.543 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:30 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:59:30.305+0000 7f45bf786700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:59.721447+0000 front 2024-04-02T04:58:59.721523+0000 (oldest deadline 2024-04-02T04:59:22.621507+0000) 2024-04-02T04:59:30.543 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:30 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:30.384+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.270997+0000 front 2024-04-02T04:58:50.271075+0000 (oldest deadline 2024-04-02T04:59:14.371167+0000) 2024-04-02T04:59:30.543 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:30 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:30.384+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:59:00.172643+0000 front 2024-04-02T04:59:00.172743+0000 (oldest deadline 2024-04-02T04:59:24.872692+0000) 2024-04-02T04:59:30.846 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:30 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:30.539+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.967123+0000 front 2024-04-02T04:58:50.967101+0000 (oldest deadline 2024-04-02T04:59:16.867341+0000) 2024-04-02T04:59:30.846 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:30 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:30.539+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:56.867795+0000 front 2024-04-02T04:58:56.867864+0000 (oldest deadline 2024-04-02T04:59:22.168002+0000) 2024-04-02T04:59:31.645 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:31 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:59:31.344+0000 7f45bf786700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:52.720639+0000 front 2024-04-02T04:58:52.720588+0000 (oldest deadline 2024-04-02T04:59:16.220734+0000) 2024-04-02T04:59:31.645 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:31 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:59:31.344+0000 7f45bf786700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:59.721447+0000 front 2024-04-02T04:58:59.721523+0000 (oldest deadline 2024-04-02T04:59:22.621507+0000) 2024-04-02T04:59:31.646 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:31 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:31.420+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.270997+0000 front 2024-04-02T04:58:50.271075+0000 (oldest deadline 2024-04-02T04:59:14.371167+0000) 2024-04-02T04:59:31.646 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:31 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:31.420+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:59:00.172643+0000 front 2024-04-02T04:59:00.172743+0000 (oldest deadline 2024-04-02T04:59:24.872692+0000) 2024-04-02T04:59:31.646 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:31 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:31.491+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.967123+0000 front 2024-04-02T04:58:50.967101+0000 (oldest deadline 2024-04-02T04:59:16.867341+0000) 2024-04-02T04:59:31.646 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:31 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:31.491+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:56.867795+0000 front 2024-04-02T04:58:56.867864+0000 (oldest deadline 2024-04-02T04:59:22.168002+0000) 2024-04-02T04:59:31.646 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:31 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:31.491+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6822 osd.2 since back 2024-04-02T04:59:09.070096+0000 front 2024-04-02T04:59:09.070239+0000 (oldest deadline 2024-04-02T04:59:31.370214+0000) 2024-04-02T04:59:32.645 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:32 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:59:32.298+0000 7f45bf786700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:52.720639+0000 front 2024-04-02T04:58:52.720588+0000 (oldest deadline 2024-04-02T04:59:16.220734+0000) 2024-04-02T04:59:32.645 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:32 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:59:32.298+0000 7f45bf786700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:59.721447+0000 front 2024-04-02T04:58:59.721523+0000 (oldest deadline 2024-04-02T04:59:22.621507+0000) 2024-04-02T04:59:32.646 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:32 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:32.454+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.270997+0000 front 2024-04-02T04:58:50.271075+0000 (oldest deadline 2024-04-02T04:59:14.371167+0000) 2024-04-02T04:59:32.646 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:32 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:32.454+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:59:00.172643+0000 front 2024-04-02T04:59:00.172743+0000 (oldest deadline 2024-04-02T04:59:24.872692+0000) 2024-04-02T04:59:32.646 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:32 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:32.524+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.967123+0000 front 2024-04-02T04:58:50.967101+0000 (oldest deadline 2024-04-02T04:59:16.867341+0000) 2024-04-02T04:59:32.646 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:32 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:32.524+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:56.867795+0000 front 2024-04-02T04:58:56.867864+0000 (oldest deadline 2024-04-02T04:59:22.168002+0000) 2024-04-02T04:59:32.646 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:32 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:32.524+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6822 osd.2 since back 2024-04-02T04:59:09.070096+0000 front 2024-04-02T04:59:09.070239+0000 (oldest deadline 2024-04-02T04:59:31.370214+0000) 2024-04-02T04:59:33.645 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:33 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:33.481+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.270997+0000 front 2024-04-02T04:58:50.271075+0000 (oldest deadline 2024-04-02T04:59:14.371167+0000) 2024-04-02T04:59:33.645 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:33 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:33.481+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:59:00.172643+0000 front 2024-04-02T04:59:00.172743+0000 (oldest deadline 2024-04-02T04:59:24.872692+0000) 2024-04-02T04:59:33.645 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:33 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:33.531+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.967123+0000 front 2024-04-02T04:58:50.967101+0000 (oldest deadline 2024-04-02T04:59:16.867341+0000) 2024-04-02T04:59:33.645 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:33 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:33.531+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:56.867795+0000 front 2024-04-02T04:58:56.867864+0000 (oldest deadline 2024-04-02T04:59:22.168002+0000) 2024-04-02T04:59:33.646 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:33 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:33.531+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6822 osd.2 since back 2024-04-02T04:59:09.070096+0000 front 2024-04-02T04:59:09.070239+0000 (oldest deadline 2024-04-02T04:59:31.370214+0000) 2024-04-02T04:59:33.646 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:33 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:59:33.300+0000 7f45bf786700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:52.720639+0000 front 2024-04-02T04:58:52.720588+0000 (oldest deadline 2024-04-02T04:59:16.220734+0000) 2024-04-02T04:59:33.646 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:33 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5[60368]: 2024-04-02T04:59:33.300+0000 7f45bf786700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:59.721447+0000 front 2024-04-02T04:58:59.721523+0000 (oldest deadline 2024-04-02T04:59:22.621507+0000) 2024-04-02T04:59:34.304 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:34 smithi122 podman[78916]: 2024-04-02 04:59:34.046030304 +0000 UTC m=+5.513367674 container died be26f425f51b969fc42f4f1a6fa98457e1c2d6884f6207536bb0ead8d1231b7e (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-6cec793, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-02T04:59:34.645 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:34 smithi122 podman[78916]: 2024-04-02 04:59:34.399397534 +0000 UTC m=+5.866734903 container cleanup be26f425f51b969fc42f4f1a6fa98457e1c2d6884f6207536bb0ead8d1231b7e (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-6cec793) 2024-04-02T04:59:34.645 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:34 smithi122 bash[78916]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5 2024-04-02T04:59:34.645 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:34 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:34.498+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.270997+0000 front 2024-04-02T04:58:50.271075+0000 (oldest deadline 2024-04-02T04:59:14.371167+0000) 2024-04-02T04:59:34.646 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:34 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:34.498+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:59:00.172643+0000 front 2024-04-02T04:59:00.172743+0000 (oldest deadline 2024-04-02T04:59:24.872692+0000) 2024-04-02T04:59:34.646 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:34 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:34.535+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.967123+0000 front 2024-04-02T04:58:50.967101+0000 (oldest deadline 2024-04-02T04:59:16.867341+0000) 2024-04-02T04:59:34.646 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:34 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:34.535+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:56.867795+0000 front 2024-04-02T04:58:56.867864+0000 (oldest deadline 2024-04-02T04:59:22.168002+0000) 2024-04-02T04:59:34.646 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:34 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:34.535+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6822 osd.2 since back 2024-04-02T04:59:09.070096+0000 front 2024-04-02T04:59:09.070239+0000 (oldest deadline 2024-04-02T04:59:31.370214+0000) 2024-04-02T04:59:35.013 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:34 smithi122 podman[78965]: 2024-04-02 04:59:34.667488902 +0000 UTC m=+0.624906128 container remove be26f425f51b969fc42f4f1a6fa98457e1c2d6884f6207536bb0ead8d1231b7e (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-6cec793, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-02T04:59:35.291 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:35 smithi122 podman[79056]: 2024-04-02 04:59:35.010061398 +0000 UTC m=+0.112230130 container create 587afe0349d5211ee36eb151c293c0143b07e9c773e44f4ebb0fb70132775e5c (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, RELEASE=squid-6cec793, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-02T04:59:35.291 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:35 smithi122 podman[79056]: 2024-04-02 04:59:34.915630341 +0000 UTC m=+0.017799073 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:59:35.291 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:35 smithi122 podman[79056]: 2024-04-02 04:59:35.204609752 +0000 UTC m=+0.306778494 container init 587afe0349d5211ee36eb151c293c0143b07e9c773e44f4ebb0fb70132775e5c (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-6cec793, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240326) 2024-04-02T04:59:35.291 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:35 smithi122 podman[79056]: 2024-04-02 04:59:35.208165166 +0000 UTC m=+0.310333902 container start 587afe0349d5211ee36eb151c293c0143b07e9c773e44f4ebb0fb70132775e5c (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5-deactivate, GIT_CLEAN=True, RELEASE=squid-6cec793, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T04:59:35.291 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:35 smithi122 podman[79056]: 2024-04-02 04:59:35.2874511 +0000 UTC m=+0.389619832 container attach 587afe0349d5211ee36eb151c293c0143b07e9c773e44f4ebb0fb70132775e5c (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5-deactivate, RELEASE=squid-6cec793, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T04:59:35.572 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 02 04:59:35 smithi122 podman[79056]: 2024-04-02 04:59:35.499324807 +0000 UTC m=+0.601493543 container died 587afe0349d5211ee36eb151c293c0143b07e9c773e44f4ebb0fb70132775e5c (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-6cec793, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux ) 2024-04-02T04:59:35.573 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:35 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:35.530+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.270997+0000 front 2024-04-02T04:58:50.271075+0000 (oldest deadline 2024-04-02T04:59:14.371167+0000) 2024-04-02T04:59:35.573 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:35 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:35.530+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:59:00.172643+0000 front 2024-04-02T04:59:00.172743+0000 (oldest deadline 2024-04-02T04:59:24.872692+0000) 2024-04-02T04:59:35.573 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:35 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:35.530+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6822 osd.2 since back 2024-04-02T04:59:09.573508+0000 front 2024-04-02T04:59:09.573540+0000 (oldest deadline 2024-04-02T04:59:35.473522+0000) 2024-04-02T04:59:35.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:35 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:35.568+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.967123+0000 front 2024-04-02T04:58:50.967101+0000 (oldest deadline 2024-04-02T04:59:16.867341+0000) 2024-04-02T04:59:35.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:35 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:35.568+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:56.867795+0000 front 2024-04-02T04:58:56.867864+0000 (oldest deadline 2024-04-02T04:59:22.168002+0000) 2024-04-02T04:59:35.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:35 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:35.568+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6822 osd.2 since back 2024-04-02T04:59:09.070096+0000 front 2024-04-02T04:59:09.070239+0000 (oldest deadline 2024-04-02T04:59:31.370214+0000) 2024-04-02T04:59:36.047 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.5.service' 2024-04-02T04:59:36.087 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T04:59:36.088 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-02T04:59:36.088 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-02T04:59:36.088 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.6 2024-04-02T04:59:36.395 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:36 smithi122 systemd[1]: Stopping Ceph osd.6 for 48e16742-f0aa-11ee-b647-cb9ed24678a4... 2024-04-02T04:59:36.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:36 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:36.543+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.967123+0000 front 2024-04-02T04:58:50.967101+0000 (oldest deadline 2024-04-02T04:59:16.867341+0000) 2024-04-02T04:59:36.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:36 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:36.543+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:56.867795+0000 front 2024-04-02T04:58:56.867864+0000 (oldest deadline 2024-04-02T04:59:22.168002+0000) 2024-04-02T04:59:36.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:36 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:36.543+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6822 osd.2 since back 2024-04-02T04:59:09.070096+0000 front 2024-04-02T04:59:09.070239+0000 (oldest deadline 2024-04-02T04:59:31.370214+0000) 2024-04-02T04:59:36.896 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:36 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:36.414+0000 7f281c7bf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T04:59:36.896 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:36 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:36.414+0000 7f281c7bf700 -1 osd.6 152 *** Got signal Terminated *** 2024-04-02T04:59:36.896 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:36 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:36.414+0000 7f281c7bf700 -1 osd.6 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T04:59:36.896 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:36 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:36.516+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.270997+0000 front 2024-04-02T04:58:50.271075+0000 (oldest deadline 2024-04-02T04:59:14.371167+0000) 2024-04-02T04:59:36.896 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:36 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:36.516+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:59:00.172643+0000 front 2024-04-02T04:59:00.172743+0000 (oldest deadline 2024-04-02T04:59:24.872692+0000) 2024-04-02T04:59:36.896 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:36 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:36.516+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6822 osd.2 since back 2024-04-02T04:59:09.573508+0000 front 2024-04-02T04:59:09.573540+0000 (oldest deadline 2024-04-02T04:59:35.473522+0000) 2024-04-02T04:59:37.895 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:37 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:37.481+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.270997+0000 front 2024-04-02T04:58:50.271075+0000 (oldest deadline 2024-04-02T04:59:14.371167+0000) 2024-04-02T04:59:37.895 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:37 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:37.481+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:59:00.172643+0000 front 2024-04-02T04:59:00.172743+0000 (oldest deadline 2024-04-02T04:59:24.872692+0000) 2024-04-02T04:59:37.895 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:37 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:37.481+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6822 osd.2 since back 2024-04-02T04:59:09.573508+0000 front 2024-04-02T04:59:09.573540+0000 (oldest deadline 2024-04-02T04:59:35.473522+0000) 2024-04-02T04:59:37.896 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:37 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:37.501+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.967123+0000 front 2024-04-02T04:58:50.967101+0000 (oldest deadline 2024-04-02T04:59:16.867341+0000) 2024-04-02T04:59:37.896 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:37 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:37.501+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:56.867795+0000 front 2024-04-02T04:58:56.867864+0000 (oldest deadline 2024-04-02T04:59:22.168002+0000) 2024-04-02T04:59:37.896 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:37 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:37.501+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6822 osd.2 since back 2024-04-02T04:59:09.070096+0000 front 2024-04-02T04:59:09.070239+0000 (oldest deadline 2024-04-02T04:59:31.370214+0000) 2024-04-02T04:59:38.895 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:38 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:38.434+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.270997+0000 front 2024-04-02T04:58:50.271075+0000 (oldest deadline 2024-04-02T04:59:14.371167+0000) 2024-04-02T04:59:38.895 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:38 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:38.434+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:59:00.172643+0000 front 2024-04-02T04:59:00.172743+0000 (oldest deadline 2024-04-02T04:59:24.872692+0000) 2024-04-02T04:59:38.896 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:38 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:38.434+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6822 osd.2 since back 2024-04-02T04:59:09.573508+0000 front 2024-04-02T04:59:09.573540+0000 (oldest deadline 2024-04-02T04:59:35.473522+0000) 2024-04-02T04:59:38.896 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:38 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:38.465+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.967123+0000 front 2024-04-02T04:58:50.967101+0000 (oldest deadline 2024-04-02T04:59:16.867341+0000) 2024-04-02T04:59:38.896 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:38 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:38.465+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:56.867795+0000 front 2024-04-02T04:58:56.867864+0000 (oldest deadline 2024-04-02T04:59:22.168002+0000) 2024-04-02T04:59:38.896 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:38 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:38.465+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6822 osd.2 since back 2024-04-02T04:59:09.070096+0000 front 2024-04-02T04:59:09.070239+0000 (oldest deadline 2024-04-02T04:59:31.370214+0000) 2024-04-02T04:59:39.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:39 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:39.429+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.967123+0000 front 2024-04-02T04:58:50.967101+0000 (oldest deadline 2024-04-02T04:59:16.867341+0000) 2024-04-02T04:59:39.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:39 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:39.429+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:56.867795+0000 front 2024-04-02T04:58:56.867864+0000 (oldest deadline 2024-04-02T04:59:22.168002+0000) 2024-04-02T04:59:39.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:39 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:39.429+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6822 osd.2 since back 2024-04-02T04:59:09.070096+0000 front 2024-04-02T04:59:09.070239+0000 (oldest deadline 2024-04-02T04:59:31.370214+0000) 2024-04-02T04:59:39.896 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:39 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:39.441+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.270997+0000 front 2024-04-02T04:58:50.271075+0000 (oldest deadline 2024-04-02T04:59:14.371167+0000) 2024-04-02T04:59:39.896 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:39 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:39.441+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:59:00.172643+0000 front 2024-04-02T04:59:00.172743+0000 (oldest deadline 2024-04-02T04:59:24.872692+0000) 2024-04-02T04:59:39.896 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:39 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:39.441+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6822 osd.2 since back 2024-04-02T04:59:09.573508+0000 front 2024-04-02T04:59:09.573540+0000 (oldest deadline 2024-04-02T04:59:35.473522+0000) 2024-04-02T04:59:39.896 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:39 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:39.441+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6830 osd.3 since back 2024-04-02T04:59:15.473988+0000 front 2024-04-02T04:59:15.473922+0000 (oldest deadline 2024-04-02T04:59:38.973962+0000) 2024-04-02T04:59:40.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:40 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:40.415+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.967123+0000 front 2024-04-02T04:58:50.967101+0000 (oldest deadline 2024-04-02T04:59:16.867341+0000) 2024-04-02T04:59:40.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:40 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:40.415+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:56.867795+0000 front 2024-04-02T04:58:56.867864+0000 (oldest deadline 2024-04-02T04:59:22.168002+0000) 2024-04-02T04:59:40.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:40 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:40.415+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6822 osd.2 since back 2024-04-02T04:59:09.070096+0000 front 2024-04-02T04:59:09.070239+0000 (oldest deadline 2024-04-02T04:59:31.370214+0000) 2024-04-02T04:59:40.895 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:40 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:40.423+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.270997+0000 front 2024-04-02T04:58:50.271075+0000 (oldest deadline 2024-04-02T04:59:14.371167+0000) 2024-04-02T04:59:40.896 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:40 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:40.423+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:59:00.172643+0000 front 2024-04-02T04:59:00.172743+0000 (oldest deadline 2024-04-02T04:59:24.872692+0000) 2024-04-02T04:59:40.896 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:40 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:40.423+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6822 osd.2 since back 2024-04-02T04:59:09.573508+0000 front 2024-04-02T04:59:09.573540+0000 (oldest deadline 2024-04-02T04:59:35.473522+0000) 2024-04-02T04:59:40.896 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:40 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:40.423+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6830 osd.3 since back 2024-04-02T04:59:15.473988+0000 front 2024-04-02T04:59:15.473922+0000 (oldest deadline 2024-04-02T04:59:38.973962+0000) 2024-04-02T04:59:41.645 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:41 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:41.385+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.270997+0000 front 2024-04-02T04:58:50.271075+0000 (oldest deadline 2024-04-02T04:59:14.371167+0000) 2024-04-02T04:59:41.645 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:41 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:41.385+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:59:00.172643+0000 front 2024-04-02T04:59:00.172743+0000 (oldest deadline 2024-04-02T04:59:24.872692+0000) 2024-04-02T04:59:41.645 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:41 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:41.385+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6822 osd.2 since back 2024-04-02T04:59:09.573508+0000 front 2024-04-02T04:59:09.573540+0000 (oldest deadline 2024-04-02T04:59:35.473522+0000) 2024-04-02T04:59:41.645 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:41 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6[62651]: 2024-04-02T04:59:41.385+0000 7f2814dab700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.53:6830 osd.3 since back 2024-04-02T04:59:15.473988+0000 front 2024-04-02T04:59:15.473922+0000 (oldest deadline 2024-04-02T04:59:38.973962+0000) 2024-04-02T04:59:41.646 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:41 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:41.395+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.967123+0000 front 2024-04-02T04:58:50.967101+0000 (oldest deadline 2024-04-02T04:59:16.867341+0000) 2024-04-02T04:59:41.646 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:41 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:41.395+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:56.867795+0000 front 2024-04-02T04:58:56.867864+0000 (oldest deadline 2024-04-02T04:59:22.168002+0000) 2024-04-02T04:59:41.646 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:41 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:41.395+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6822 osd.2 since back 2024-04-02T04:59:09.070096+0000 front 2024-04-02T04:59:09.070239+0000 (oldest deadline 2024-04-02T04:59:31.370214+0000) 2024-04-02T04:59:41.646 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:41 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:41.395+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6830 osd.3 since back 2024-04-02T04:59:16.571639+0000 front 2024-04-02T04:59:16.571600+0000 (oldest deadline 2024-04-02T04:59:41.271746+0000) 2024-04-02T04:59:42.113 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:41 smithi122 podman[79162]: 2024-04-02 04:59:41.802708786 +0000 UTC m=+5.493801304 container died aab18af876da2ac5f684ac887350f6af9b7d61cd6173a57f1613cddb7989a3f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-6cec793, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T04:59:42.113 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:42 smithi122 podman[79162]: 2024-04-02 04:59:42.110002031 +0000 UTC m=+5.801094553 container cleanup aab18af876da2ac5f684ac887350f6af9b7d61cd6173a57f1613cddb7989a3f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-6cec793, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-02T04:59:42.395 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:42 smithi122 bash[79162]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6 2024-04-02T04:59:42.395 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:42 smithi122 podman[79179]: 2024-04-02 04:59:42.344882634 +0000 UTC m=+0.542130693 container remove aab18af876da2ac5f684ac887350f6af9b7d61cd6173a57f1613cddb7989a3f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-6cec793, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-02T04:59:42.767 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:42 smithi122 podman[79267]: 2024-04-02 04:59:42.552497308 +0000 UTC m=+0.018102820 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:59:42.768 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:42 smithi122 podman[79267]: 2024-04-02 04:59:42.662206283 +0000 UTC m=+0.127811768 container create 6f70458b8654d081caa0de352b0a067e885da4c5ea4c66d98fd6a5d63cf1f564 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-6cec793, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T04:59:42.768 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:42 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:42.413+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.967123+0000 front 2024-04-02T04:58:50.967101+0000 (oldest deadline 2024-04-02T04:59:16.867341+0000) 2024-04-02T04:59:42.768 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:42 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:42.413+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:56.867795+0000 front 2024-04-02T04:58:56.867864+0000 (oldest deadline 2024-04-02T04:59:22.168002+0000) 2024-04-02T04:59:42.768 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:42 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:42.413+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6822 osd.2 since back 2024-04-02T04:59:09.070096+0000 front 2024-04-02T04:59:09.070239+0000 (oldest deadline 2024-04-02T04:59:31.370214+0000) 2024-04-02T04:59:42.768 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:42 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:42.413+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6830 osd.3 since back 2024-04-02T04:59:16.571639+0000 front 2024-04-02T04:59:16.571600+0000 (oldest deadline 2024-04-02T04:59:41.271746+0000) 2024-04-02T04:59:43.145 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:42 smithi122 podman[79267]: 2024-04-02 04:59:42.864983285 +0000 UTC m=+0.330588794 container init 6f70458b8654d081caa0de352b0a067e885da4c5ea4c66d98fd6a5d63cf1f564 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-6cec793) 2024-04-02T04:59:43.145 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:42 smithi122 podman[79267]: 2024-04-02 04:59:42.868366115 +0000 UTC m=+0.333971600 container start 6f70458b8654d081caa0de352b0a067e885da4c5ea4c66d98fd6a5d63cf1f564 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-6cec793, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T04:59:43.145 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:42 smithi122 podman[79267]: 2024-04-02 04:59:42.947920877 +0000 UTC m=+0.413526365 container attach 6f70458b8654d081caa0de352b0a067e885da4c5ea4c66d98fd6a5d63cf1f564 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6-deactivate, RELEASE=squid-6cec793, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-02T04:59:43.454 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 02 04:59:43 smithi122 podman[79267]: 2024-04-02 04:59:43.161460046 +0000 UTC m=+0.627065546 container died 6f70458b8654d081caa0de352b0a067e885da4c5ea4c66d98fd6a5d63cf1f564 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-6-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, RELEASE=squid-6cec793, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-02T04:59:43.705 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:43 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:43.454+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.967123+0000 front 2024-04-02T04:58:50.967101+0000 (oldest deadline 2024-04-02T04:59:16.867341+0000) 2024-04-02T04:59:43.705 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:43 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:43.455+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:56.867795+0000 front 2024-04-02T04:58:56.867864+0000 (oldest deadline 2024-04-02T04:59:22.168002+0000) 2024-04-02T04:59:43.705 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:43 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:43.455+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6822 osd.2 since back 2024-04-02T04:59:09.070096+0000 front 2024-04-02T04:59:09.070239+0000 (oldest deadline 2024-04-02T04:59:31.370214+0000) 2024-04-02T04:59:43.705 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:43 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:43.455+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6830 osd.3 since back 2024-04-02T04:59:16.571639+0000 front 2024-04-02T04:59:16.571600+0000 (oldest deadline 2024-04-02T04:59:41.271746+0000) 2024-04-02T04:59:43.724 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.6.service' 2024-04-02T04:59:43.765 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T04:59:43.765 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-02T04:59:43.765 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-02T04:59:43.766 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.7 2024-04-02T04:59:44.083 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:43 smithi122 systemd[1]: Stopping Ceph osd.7 for 48e16742-f0aa-11ee-b647-cb9ed24678a4... 2024-04-02T04:59:44.395 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:44 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:44.079+0000 7f8923344700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T04:59:44.395 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:44 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:44.079+0000 7f8923344700 -1 osd.7 152 *** Got signal Terminated *** 2024-04-02T04:59:44.395 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:44 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:44.079+0000 7f8923344700 -1 osd.7 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T04:59:44.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:44 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:44.420+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.967123+0000 front 2024-04-02T04:58:50.967101+0000 (oldest deadline 2024-04-02T04:59:16.867341+0000) 2024-04-02T04:59:44.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:44 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:44.420+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:56.867795+0000 front 2024-04-02T04:58:56.867864+0000 (oldest deadline 2024-04-02T04:59:22.168002+0000) 2024-04-02T04:59:44.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:44 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:44.420+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6822 osd.2 since back 2024-04-02T04:59:09.070096+0000 front 2024-04-02T04:59:09.070239+0000 (oldest deadline 2024-04-02T04:59:31.370214+0000) 2024-04-02T04:59:44.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:44 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:44.420+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6830 osd.3 since back 2024-04-02T04:59:16.571639+0000 front 2024-04-02T04:59:16.571600+0000 (oldest deadline 2024-04-02T04:59:41.271746+0000) 2024-04-02T04:59:45.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:45 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:45.394+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.967123+0000 front 2024-04-02T04:58:50.967101+0000 (oldest deadline 2024-04-02T04:59:16.867341+0000) 2024-04-02T04:59:45.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:45 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:45.394+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:56.867795+0000 front 2024-04-02T04:58:56.867864+0000 (oldest deadline 2024-04-02T04:59:22.168002+0000) 2024-04-02T04:59:45.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:45 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:45.394+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6822 osd.2 since back 2024-04-02T04:59:09.070096+0000 front 2024-04-02T04:59:09.070239+0000 (oldest deadline 2024-04-02T04:59:31.370214+0000) 2024-04-02T04:59:45.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:45 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:45.394+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6830 osd.3 since back 2024-04-02T04:59:16.571639+0000 front 2024-04-02T04:59:16.571600+0000 (oldest deadline 2024-04-02T04:59:41.271746+0000) 2024-04-02T04:59:46.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:46 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:46.434+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.967123+0000 front 2024-04-02T04:58:50.967101+0000 (oldest deadline 2024-04-02T04:59:16.867341+0000) 2024-04-02T04:59:46.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:46 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:46.434+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:56.867795+0000 front 2024-04-02T04:58:56.867864+0000 (oldest deadline 2024-04-02T04:59:22.168002+0000) 2024-04-02T04:59:46.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:46 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:46.434+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6822 osd.2 since back 2024-04-02T04:59:09.070096+0000 front 2024-04-02T04:59:09.070239+0000 (oldest deadline 2024-04-02T04:59:31.370214+0000) 2024-04-02T04:59:46.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:46 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:46.434+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6830 osd.3 since back 2024-04-02T04:59:16.571639+0000 front 2024-04-02T04:59:16.571600+0000 (oldest deadline 2024-04-02T04:59:41.271746+0000) 2024-04-02T04:59:47.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:47 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:47.441+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.967123+0000 front 2024-04-02T04:58:50.967101+0000 (oldest deadline 2024-04-02T04:59:16.867341+0000) 2024-04-02T04:59:47.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:47 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:47.441+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:56.867795+0000 front 2024-04-02T04:58:56.867864+0000 (oldest deadline 2024-04-02T04:59:22.168002+0000) 2024-04-02T04:59:47.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:47 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:47.441+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6822 osd.2 since back 2024-04-02T04:59:09.070096+0000 front 2024-04-02T04:59:09.070239+0000 (oldest deadline 2024-04-02T04:59:31.370214+0000) 2024-04-02T04:59:47.895 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:47 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:47.441+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6830 osd.3 since back 2024-04-02T04:59:16.571639+0000 front 2024-04-02T04:59:16.571600+0000 (oldest deadline 2024-04-02T04:59:41.271746+0000) 2024-04-02T04:59:48.796 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:48 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:48.484+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2024-04-02T04:58:50.967123+0000 front 2024-04-02T04:58:50.967101+0000 (oldest deadline 2024-04-02T04:59:16.867341+0000) 2024-04-02T04:59:48.797 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:48 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:48.484+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6814 osd.1 since back 2024-04-02T04:58:56.867795+0000 front 2024-04-02T04:58:56.867864+0000 (oldest deadline 2024-04-02T04:59:22.168002+0000) 2024-04-02T04:59:48.797 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:48 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:48.484+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6822 osd.2 since back 2024-04-02T04:59:09.070096+0000 front 2024-04-02T04:59:09.070239+0000 (oldest deadline 2024-04-02T04:59:31.370214+0000) 2024-04-02T04:59:48.797 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:48 smithi122 ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7[64921]: 2024-04-02T04:59:48.484+0000 7f891b930700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.53:6830 osd.3 since back 2024-04-02T04:59:16.571639+0000 front 2024-04-02T04:59:16.571600+0000 (oldest deadline 2024-04-02T04:59:41.271746+0000) 2024-04-02T04:59:49.909 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:49 smithi122 podman[79373]: 2024-04-02 04:59:49.646297476 +0000 UTC m=+5.665682790 container died 86ee236f94268657c18e051249afa168e95cce7a09ae51108098f60e0fd9ab82 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7, RELEASE=squid-6cec793, org.label-schema.build-date=20240326, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-02T04:59:50.258 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:50 smithi122 podman[79373]: 2024-04-02 04:59:50.012488666 +0000 UTC m=+6.031873979 container cleanup 86ee236f94268657c18e051249afa168e95cce7a09ae51108098f60e0fd9ab82 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-6cec793, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-02T04:59:50.258 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:50 smithi122 bash[79373]: ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7 2024-04-02T04:59:50.567 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:50 smithi122 podman[79401]: 2024-04-02 04:59:50.255135424 +0000 UTC m=+0.606060105 container remove 86ee236f94268657c18e051249afa168e95cce7a09ae51108098f60e0fd9ab82 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-6cec793, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-02T04:59:50.820 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:50 smithi122 podman[79493]: 2024-04-02 04:59:50.564386263 +0000 UTC m=+0.112250995 container create 01b1d0a2df606b9d8ff44c44cab58b9439e29efa32fe1fb42a4dde930b6b2e28 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7-deactivate, RELEASE=squid-6cec793, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-02T04:59:50.820 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:50 smithi122 podman[79493]: 2024-04-02 04:59:50.469569611 +0000 UTC m=+0.017434290 image pull 78f830190839e38cac46d0f24741058bd5f6b4363a488caf8a9e060d9857e808 quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae 2024-04-02T04:59:50.820 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:50 smithi122 podman[79493]: 2024-04-02 04:59:50.758214272 +0000 UTC m=+0.306078948 container init 01b1d0a2df606b9d8ff44c44cab58b9439e29efa32fe1fb42a4dde930b6b2e28 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-6cec793, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-02T04:59:50.820 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:50 smithi122 podman[79493]: 2024-04-02 04:59:50.76148436 +0000 UTC m=+0.309349027 container start 01b1d0a2df606b9d8ff44c44cab58b9439e29efa32fe1fb42a4dde930b6b2e28 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-6cec793, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-02T04:59:51.145 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:50 smithi122 podman[79493]: 2024-04-02 04:59:50.81655515 +0000 UTC m=+0.364419830 container attach 01b1d0a2df606b9d8ff44c44cab58b9439e29efa32fe1fb42a4dde930b6b2e28 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, RELEASE=squid-6cec793, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T04:59:51.145 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 02 04:59:51 smithi122 podman[79493]: 2024-04-02 04:59:51.052615771 +0000 UTC m=+0.600480447 container died 01b1d0a2df606b9d8ff44c44cab58b9439e29efa32fe1fb42a4dde930b6b2e28 (image=quay.ceph.io/ceph-ci/ceph@sha256:7bd5b84926d4a0d6c51e5562e965bff211cec2740c9f4dec9b5e00a2bbb75eae, name=ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4-osd-7-deactivate, ceph=True, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-6cec793, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T04:59:51.704 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48e16742-f0aa-11ee-b647-cb9ed24678a4@osd.7.service' 2024-04-02T04:59:51.746 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T04:59:51.747 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-02T04:59:51.747 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 --force --keep-logs 2024-04-02T05:02:47.840 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 --force --keep-logs 2024-04-02T05:03:07.612 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-02T05:03:07.648 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-02T05:03:07.677 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-02T05:03:07.679 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634849/remote/smithi053/crash 2024-04-02T05:03:07.679 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/crash -- . 2024-04-02T05:03:07.736 INFO:teuthology.orchestra.run.smithi053.stderr:tar: /var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-04-02T05:03:07.737 INFO:teuthology.orchestra.run.smithi053.stderr:tar: Error is not recoverable: exiting now 2024-04-02T05:03:07.738 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634849/remote/smithi122/crash 2024-04-02T05:03:07.739 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/crash -- . 2024-04-02T05:03:07.773 INFO:teuthology.orchestra.run.smithi122.stderr:tar: /var/lib/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-04-02T05:03:07.773 INFO:teuthology.orchestra.run.smithi122.stderr:tar: Error is not recoverable: exiting now 2024-04-02T05:03:07.774 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-02T05:03:07.775 DEBUG:teuthology.orchestra.run.smithi053:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-02T05:03:07.806 INFO:teuthology.orchestra.run.smithi053.stdout:2024-04-02T04:46:49.936349+0000 mon.smithi053 (mon.0) 872 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-02T05:03:07.806 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-02T05:03:07.806 DEBUG:teuthology.orchestra.run.smithi053:> sudo egrep '\[SEC\]' /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-02T05:03:07.876 DEBUG:teuthology.orchestra.run.smithi053:> sudo egrep '\[ERR\]' /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-02T05:03:07.946 DEBUG:teuthology.orchestra.run.smithi053:> sudo egrep '\[WRN\]' /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-02T05:03:08.019 INFO:teuthology.orchestra.run.smithi053.stdout:2024-04-02T04:46:49.936349+0000 mon.smithi053 (mon.0) 872 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-02T05:03:08.019 INFO:tasks.cephadm:Compressing logs... 2024-04-02T05:03:08.019 DEBUG:teuthology.orchestra.run.smithi053:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-02T05:03:08.063 DEBUG:teuthology.orchestra.run.smithi122:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-02T05:03:08.087 INFO:teuthology.orchestra.run.smithi053.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-02T05:03:08.087 INFO:teuthology.orchestra.run.smithi053.stderr:: No such file or directory 2024-04-02T05:03:08.087 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-mgr.smithi053.legdmv.log 2024-04-02T05:03:08.087 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph.audit.log 2024-04-02T05:03:08.089 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-mgr.smithi053.legdmv.log: gzip -5 --verbose -- /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-osd.1.log 2024-04-02T05:03:08.089 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-osd.3.log 2024-04-02T05:03:08.089 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-mon.smithi053.log 2024-04-02T05:03:08.091 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-osd.1.log: /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-osd.0.log 2024-04-02T05:03:08.091 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-mon.smithi053.log: gzip -5 --verbose -- /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph.cephadm.log 2024-04-02T05:03:08.091 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph.log 2024-04-02T05:03:08.091 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph.cephadm.log: 85.6% -- replaced with /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph.cephadm.log.gz 2024-04-02T05:03:08.092 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-osd.2.log 2024-04-02T05:03:08.093 INFO:teuthology.orchestra.run.smithi122.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-02T05:03:08.093 INFO:teuthology.orchestra.run.smithi122.stderr:: No such file or directory 2024-04-02T05:03:08.093 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-osd.6.log 2024-04-02T05:03:08.093 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-mon.smithi122.log 2024-04-02T05:03:08.093 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph.audit.log 2024-04-02T05:03:08.093 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-osd.4.log 2024-04-02T05:03:08.093 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-mon.smithi122.log: gzip -5 --verbose -- /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-osd.7.log 2024-04-02T05:03:08.094 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph.log: 91.5% -- replaced with /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph.audit.log.gz 2024-04-02T05:03:08.099 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph.audit.log: /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-osd.5.log 2024-04-02T05:03:08.099 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-mds.foofs.smithi122.mwoyoh.log 2024-04-02T05:03:08.099 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph.cephadm.log 2024-04-02T05:03:08.100 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-mds.foofs.smithi122.mwoyoh.log: 88.4% -- replaced with /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-mds.foofs.smithi122.mwoyoh.log.gz 2024-04-02T05:03:08.100 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph.log 2024-04-02T05:03:08.100 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph.cephadm.log: 85.7% -- replaced with /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph.cephadm.log.gz 2024-04-02T05:03:08.100 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-volume.log 2024-04-02T05:03:08.100 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph.log: 91.6% -- replaced with /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph.audit.log.gz 2024-04-02T05:03:08.100 INFO:teuthology.orchestra.run.smithi122.stderr: 90.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-02T05:03:08.100 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-mgr.smithi122.cpwlyk.log 2024-04-02T05:03:08.100 INFO:teuthology.orchestra.run.smithi053.stderr: 87.5% -- replaced with /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph.log.gz 2024-04-02T05:03:08.100 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-volume.log 2024-04-02T05:03:08.100 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-mds.foofs.smithi053.nfrfot.log 2024-04-02T05:03:08.101 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-volume.log: /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-mgr.smithi122.cpwlyk.log: 87.5% -- replaced with /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph.log.gz 2024-04-02T05:03:08.113 INFO:teuthology.orchestra.run.smithi122.stderr: 91.5% -- replaced with /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-volume.log.gz 2024-04-02T05:03:08.125 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-volume.log: /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-mds.foofs.smithi053.nfrfot.log: 91.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-02T05:03:08.129 INFO:teuthology.orchestra.run.smithi053.stderr: 91.4% -- replaced with /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-volume.log.gz 2024-04-02T05:03:08.137 INFO:teuthology.orchestra.run.smithi053.stderr: 91.5% -- replaced with /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-mds.foofs.smithi053.nfrfot.log.gz 2024-04-02T05:03:08.169 INFO:teuthology.orchestra.run.smithi122.stderr: 89.8% -- replaced with /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-mgr.smithi122.cpwlyk.log.gz 2024-04-02T05:03:08.419 INFO:teuthology.orchestra.run.smithi122.stderr: 92.6% -- replaced with /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-mon.smithi122.log.gz 2024-04-02T05:03:08.501 INFO:teuthology.orchestra.run.smithi053.stderr: 88.3% -- replaced with /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-mgr.smithi053.legdmv.log.gz 2024-04-02T05:03:10.103 INFO:teuthology.orchestra.run.smithi053.stderr: 91.9% -- replaced with /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-mon.smithi053.log.gz 2024-04-02T05:03:22.348 INFO:teuthology.orchestra.run.smithi053.stderr: 94.4% -- replaced with /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-osd.2.log.gz 2024-04-02T05:03:23.065 INFO:teuthology.orchestra.run.smithi122.stderr: 93.9% -- replaced with /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-osd.4.log.gz 2024-04-02T05:03:23.596 INFO:teuthology.orchestra.run.smithi122.stderr: 94.2% -- replaced with /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-osd.6.log.gz 2024-04-02T05:03:25.403 INFO:teuthology.orchestra.run.smithi053.stderr: 94.1% -- replaced with /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-osd.0.log.gz 2024-04-02T05:03:26.789 INFO:teuthology.orchestra.run.smithi053.stderr: 94.2% -- replaced with /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-osd.1.log.gz 2024-04-02T05:03:27.506 INFO:teuthology.orchestra.run.smithi122.stderr: 94.2% -- replaced with /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-osd.5.log.gz 2024-04-02T05:03:28.228 INFO:teuthology.orchestra.run.smithi053.stderr: 94.1% -- replaced with /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-osd.3.log.gz 2024-04-02T05:03:28.229 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-04-02T05:03:28.230 INFO:teuthology.orchestra.run.smithi053.stderr:real 0m20.155s 2024-04-02T05:03:28.230 INFO:teuthology.orchestra.run.smithi053.stderr:user 1m10.262s 2024-04-02T05:03:28.230 INFO:teuthology.orchestra.run.smithi053.stderr:sys 0m2.552s 2024-04-02T05:03:32.372 INFO:teuthology.orchestra.run.smithi122.stderr: 94.0% -- replaced with /var/log/ceph/48e16742-f0aa-11ee-b647-cb9ed24678a4/ceph-osd.7.log.gz 2024-04-02T05:03:32.374 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-02T05:03:32.374 INFO:teuthology.orchestra.run.smithi122.stderr:real 0m24.296s 2024-04-02T05:03:32.374 INFO:teuthology.orchestra.run.smithi122.stderr:user 1m11.661s 2024-04-02T05:03:32.374 INFO:teuthology.orchestra.run.smithi122.stderr:sys 0m2.827s 2024-04-02T05:03:32.374 INFO:tasks.cephadm:Archiving logs... 2024-04-02T05:03:32.375 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634849/remote/smithi053/log 2024-04-02T05:03:32.376 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-02T05:03:41.737 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634849/remote/smithi122/log 2024-04-02T05:03:41.738 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-02T05:03:52.200 INFO:tasks.cephadm:Removing cluster... 2024-04-02T05:03:52.200 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 --force 2024-04-02T05:03:52.579 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 48e16742-f0aa-11ee-b647-cb9ed24678a4 --force 2024-04-02T05:03:52.967 INFO:tasks.cephadm:Removing cephadm ... 2024-04-02T05:03:52.968 DEBUG:teuthology.orchestra.run.smithi053:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-02T05:03:52.983 DEBUG:teuthology.orchestra.run.smithi122:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-02T05:03:52.997 INFO:tasks.cephadm:Teardown complete 2024-04-02T05:03:52.998 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-02T05:03:53.012 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-02T05:03:53.012 DEBUG:teuthology.orchestra.run.smithi053:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-02T05:03:53.025 DEBUG:teuthology.orchestra.run.smithi122:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-02T05:03:53.037 INFO:teuthology.orchestra.run.smithi053.stderr:bash: line 1: ntpq: command not found 2024-04-02T05:03:53.041 INFO:teuthology.orchestra.run.smithi053.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-02T05:03:53.042 INFO:teuthology.orchestra.run.smithi053.stdout:=============================================================================== 2024-04-02T05:03:53.042 INFO:teuthology.orchestra.run.smithi053.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 16 -1291us[-1262us] +/- 72ms 2024-04-02T05:03:53.042 INFO:teuthology.orchestra.run.smithi053.stdout:^* hv02.front.sepia.ceph.com 3 6 377 15 -2458us[-2429us] +/- 68ms 2024-04-02T05:03:53.042 INFO:teuthology.orchestra.run.smithi053.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 15 +5060us[+5060us] +/- 92ms 2024-04-02T05:03:53.042 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T05:03:53.051 INFO:teuthology.orchestra.run.smithi122.stderr:bash: line 1: ntpq: command not found 2024-04-02T05:03:53.055 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-02T05:03:53.056 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2024-04-02T05:03:53.056 INFO:teuthology.orchestra.run.smithi122.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 82 -1259us[-1156us] +/- 71ms 2024-04-02T05:03:53.056 INFO:teuthology.orchestra.run.smithi122.stdout:^* hv02.front.sepia.ceph.com 3 6 377 13 -2471us[-2358us] +/- 68ms 2024-04-02T05:03:53.056 INFO:teuthology.orchestra.run.smithi122.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 15 +5009us[+5122us] +/- 92ms 2024-04-02T05:03:53.056 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T05:03:53.056 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-02T05:03:53.072 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-02T05:03:53.073 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-02T05:03:53.085 DEBUG:teuthology.orchestra.run.smithi053:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-02T05:03:53.170 DEBUG:teuthology.orchestra.run.smithi122:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-02T05:03:53.248 DEBUG:teuthology.orchestra.run.smithi053:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-02T05:03:53.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:53.278 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-02T05:03:53.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:03:53.308 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-02T05:03:53.320 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-02T05:03:53.332 INFO:teuthology.task.internal:Duration was 2034.143333 seconds 2024-04-02T05:03:53.333 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-02T05:03:53.347 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-02T05:03:53.348 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-02T05:03:53.350 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-02T05:03:53.391 INFO:teuthology.orchestra.run.smithi053.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-02T05:03:53.393 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-02T05:03:53.849 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-02T05:03:53.850 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi053.front.sepia.ceph.com 2024-04-02T05:03:53.850 DEBUG:teuthology.orchestra.run.smithi053:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-02T05:03:53.911 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi122.front.sepia.ceph.com 2024-04-02T05:03:53.954 DEBUG:teuthology.orchestra.run.smithi122:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-02T05:03:54.005 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-02T05:03:54.005 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-02T05:03:54.008 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-02T05:03:54.078 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-02T05:03:54.079 DEBUG:teuthology.orchestra.run.smithi053:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-02T05:03:54.083 DEBUG:teuthology.orchestra.run.smithi122:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-02T05:04:07.838 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-02T05:04:07.850 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-02T05:04:07.850 DEBUG:teuthology.orchestra.run.smithi053:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-02T05:04:07.876 DEBUG:teuthology.orchestra.run.smithi122:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-02T05:04:07.908 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-02T05:04:07.921 DEBUG:teuthology.orchestra.run.smithi053:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-02T05:04:07.924 DEBUG:teuthology.orchestra.run.smithi122:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-02T05:04:07.946 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = core 2024-04-02T05:04:07.976 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = core 2024-04-02T05:04:08.072 DEBUG:teuthology.orchestra.run.smithi053:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-02T05:04:08.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:04:08.086 DEBUG:teuthology.orchestra.run.smithi122:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-02T05:04:08.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T05:04:08.127 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-02T05:04:08.138 INFO:teuthology.task.internal:Transferring archived files... 2024-04-02T05:04:08.139 DEBUG:teuthology.misc:Transferring archived files from smithi053:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634849/remote/smithi053 2024-04-02T05:04:08.139 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-02T05:04:08.205 DEBUG:teuthology.misc:Transferring archived files from smithi122:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634849/remote/smithi122 2024-04-02T05:04:08.206 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-02T05:04:08.249 INFO:teuthology.task.internal:Removing archive directory... 2024-04-02T05:04:08.250 DEBUG:teuthology.orchestra.run.smithi053:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-02T05:04:08.252 DEBUG:teuthology.orchestra.run.smithi122:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-02T05:04:08.297 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-02T05:04:08.310 INFO:teuthology.task.internal:Not uploading archives. 2024-04-02T05:04:08.310 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-02T05:04:08.327 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-02T05:04:08.328 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-02T05:04:08.330 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-02T05:04:08.342 INFO:teuthology.orchestra.run.smithi053.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 2 05:04 /home/ubuntu/cephtest 2024-04-02T05:04:08.351 INFO:teuthology.orchestra.run.smithi122.stdout: 266042 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 2 05:04 /home/ubuntu/cephtest 2024-04-02T05:04:08.352 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-02T05:04:08.365 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-02T05:04:08.395 INFO:teuthology.nuke:Checking targets against current locks 2024-04-02T05:04:08.423 DEBUG:teuthology.nuke:shortname: smithi053 2024-04-02T05:04:08.423 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-02T05:04:08.448 DEBUG:teuthology.nuke:shortname: smithi122 2024-04-02T05:04:08.448 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-02T05:04:08.452 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634849', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-02 04:21:54.924375', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB6lGwmTurR8jDyMJ7pg92j/x13b4nh/4eyDxRLVKZZPOw1+Or1d1EcRGjSwjo1rtDinHJ2GPyuDYw7MuajOFjY='} 2024-04-02T05:04:08.471 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-01_21:08:08-orch-squid-distro-default-smithi/7634849', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-02 04:21:54.925808', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPH70r4gDZkVgtnf+XQEzpooFGq8G2FD8nASXwlDtS7Z7n55T3x+t3yEDXeqiB1V862PNVOszCjW6l+gUgJiLiE='} 2024-04-02T05:04:08.502 INFO:teuthology.orchestra.console.smithi053:Power off 2024-04-02T05:04:08.503 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-02T05:04:08.523 INFO:teuthology.orchestra.console.smithi122:Power off 2024-04-02T05:04:08.523 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-02T05:04:08.533 DEBUG:teuthology.orchestra.console.smithi053:power off output: Chassis Power Control: Down/Off 2024-04-02T05:04:08.533 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T05:04:08.545 DEBUG:teuthology.orchestra.console.smithi122:power off output: Chassis Power Control: Down/Off 2024-04-02T05:04:08.546 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T05:04:08.557 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2024-04-02T05:04:08.568 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is on 2024-04-02T05:04:12.558 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T05:04:12.569 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T05:04:12.669 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2024-04-02T05:04:12.678 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is on 2024-04-02T05:04:16.675 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T05:04:16.686 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T05:04:16.789 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-04-02T05:04:16.799 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is off 2024-04-02T05:04:16.890 INFO:teuthology.orchestra.console.smithi053:Power off completed 2024-04-02T05:04:16.900 INFO:teuthology.orchestra.console.smithi122:Power off completed 2024-04-02T05:04:17.005 INFO:teuthology.run:Summary data: description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} duration: 2034.1433327198029 failure_reason: '"2024-04-02T04:46:49.936349+0000 mon.smithi053 (mon.0) 872 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-04-02T05:04:17.006 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-02T05:04:17.090 INFO:teuthology.run:FAIL