2024-03-26T01:36:37.588 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-26T01:36:37.589 DEBUG:teuthology.run:Teuthology command: teuthology --name teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi --owner scheduled_teuthology@teuthology --verbose --description orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} --archive /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621920 -- /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621920/orig.config.yaml 2024-03-26T01:36:37.629 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T01:36:37.710 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621920 branch: squid description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} email: ceph-qa@ceph.com first_in_suite: false job_id: '7621920' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 workunit: branch: squid sha1: 9f04c91b77e4619963d41730d8cf756e3a6152de owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - host.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - host.b - osd.4 - osd.5 - osd.6 - osd.7 seed: 1822 sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a sleep_before_teardown: 0 subset: 53/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 9f04c91b77e4619963d41730d8cf756e3a6152de targets: smithi019.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNARUXwtGAtTikQf6mQLs0++GTP3mUSGmJkGxI2L3YK/8DemFBOgWGq4CI1bAzv3ro4KJ6LTu9PWtnI8awUBNCg= smithi069.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN/SSNXMqbRRu6yK4qoW+u8KiihIo/Fuyw+tGvc1HH69D5i259F5aRCmQNaJakPSwhwY2Q5EeEJNe6sNQ9clQXI= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 roleless: true - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - vip.exec: all-hosts: - systemctl stop nfs-server - cephadm.shell: host.a: - ceph fs volume create foofs - cephadm.wait_for_service: service: mds.foofs - cephadm.shell: host.a: - ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2 - ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake - while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done - vip.exec: host.a: - mkdir /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done - echo test > /mnt/foo/testfile - sync - parallel: - upgrade-tasks - workload-tasks - vip.exec: host.a: - umount /mnt/foo - cephadm.shell: host.a: - ceph nfs cluster ls | grep foo - ceph nfs export ls foo --detailed - rados -p .nfs --all ls - - cephadm.shell: host.a: - 'set -ex [[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]] ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_21:08:11 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph mgr module enable nfs --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - cephadm.wait_for_service: service: nfs.foo user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - exec: host.a: - cd /mnt/foo && dbench 5 -t 600 || true - umount /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done - cd /mnt/foo && dbench 5 -t 5 2024-03-26T01:36:37.711 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa; will attempt to use it 2024-03-26T01:36:37.711 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa/tasks 2024-03-26T01:36:37.711 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-26T01:36:37.713 INFO:teuthology.task.internal:Checking packages... 2024-03-26T01:36:37.739 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5e70124b46c2a671efcaae4690cd9ea72670710a' 2024-03-26T01:36:37.739 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-26T01:36:37.739 INFO:teuthology.packaging:ref: None 2024-03-26T01:36:37.739 INFO:teuthology.packaging:tag: None 2024-03-26T01:36:37.740 INFO:teuthology.packaging:branch: squid 2024-03-26T01:36:37.740 INFO:teuthology.packaging:sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T01:36:37.740 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-26T01:36:37.895 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1863.g9f04c91b 2024-03-26T01:36:37.897 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-26T01:36:37.903 INFO:teuthology.task.internal:no buildpackages task found 2024-03-26T01:36:37.904 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-26T01:36:37.909 INFO:teuthology.task.internal:Saving configuration 2024-03-26T01:36:37.923 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-26T01:36:37.929 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T01:36:37.955 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi019.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621920', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 01:31:52.202525', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNARUXwtGAtTikQf6mQLs0++GTP3mUSGmJkGxI2L3YK/8DemFBOgWGq4CI1bAzv3ro4KJ6LTu9PWtnI8awUBNCg='} 2024-03-26T01:36:37.975 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621920', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 01:31:52.204192', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN/SSNXMqbRRu6yK4qoW+u8KiihIo/Fuyw+tGvc1HH69D5i259F5aRCmQNaJakPSwhwY2Q5EeEJNe6sNQ9clQXI='} 2024-03-26T01:36:37.975 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-26T01:36:37.981 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-03-26T01:36:37.982 INFO:teuthology.task.internal:roles: ubuntu@smithi069.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-03-26T01:36:37.982 INFO:teuthology.run_tasks:Running task console_log... 2024-03-26T01:36:38.055 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3296395f70>, signals=[15]) 2024-03-26T01:36:38.055 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-26T01:36:38.061 INFO:teuthology.task.internal:Opening connections... 2024-03-26T01:36:38.061 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi019.front.sepia.ceph.com 2024-03-26T01:36:38.064 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:36:38.170 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi069.front.sepia.ceph.com 2024-03-26T01:36:38.171 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:36:38.261 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-26T01:36:38.270 DEBUG:teuthology.orchestra.run.smithi019:> uname -m 2024-03-26T01:36:38.287 INFO:teuthology.orchestra.run.smithi019.stdout:x86_64 2024-03-26T01:36:38.287 DEBUG:teuthology.orchestra.run.smithi019:> cat /etc/os-release 2024-03-26T01:36:38.344 INFO:teuthology.orchestra.run.smithi019.stdout:NAME="CentOS Stream" 2024-03-26T01:36:38.344 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION="9" 2024-03-26T01:36:38.345 INFO:teuthology.orchestra.run.smithi019.stdout:ID="centos" 2024-03-26T01:36:38.345 INFO:teuthology.orchestra.run.smithi019.stdout:ID_LIKE="rhel fedora" 2024-03-26T01:36:38.345 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION_ID="9" 2024-03-26T01:36:38.345 INFO:teuthology.orchestra.run.smithi019.stdout:PLATFORM_ID="platform:el9" 2024-03-26T01:36:38.345 INFO:teuthology.orchestra.run.smithi019.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T01:36:38.345 INFO:teuthology.orchestra.run.smithi019.stdout:ANSI_COLOR="0;31" 2024-03-26T01:36:38.345 INFO:teuthology.orchestra.run.smithi019.stdout:LOGO="fedora-logo-icon" 2024-03-26T01:36:38.345 INFO:teuthology.orchestra.run.smithi019.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T01:36:38.345 INFO:teuthology.orchestra.run.smithi019.stdout:HOME_URL="https://centos.org/" 2024-03-26T01:36:38.345 INFO:teuthology.orchestra.run.smithi019.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T01:36:38.345 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T01:36:38.345 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T01:36:38.346 INFO:teuthology.lock.ops:Updating smithi019.front.sepia.ceph.com on lock server 2024-03-26T01:36:38.379 DEBUG:teuthology.orchestra.run.smithi069:> uname -m 2024-03-26T01:36:38.396 INFO:teuthology.orchestra.run.smithi069.stdout:x86_64 2024-03-26T01:36:38.397 DEBUG:teuthology.orchestra.run.smithi069:> cat /etc/os-release 2024-03-26T01:36:38.453 INFO:teuthology.orchestra.run.smithi069.stdout:NAME="CentOS Stream" 2024-03-26T01:36:38.453 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION="9" 2024-03-26T01:36:38.453 INFO:teuthology.orchestra.run.smithi069.stdout:ID="centos" 2024-03-26T01:36:38.453 INFO:teuthology.orchestra.run.smithi069.stdout:ID_LIKE="rhel fedora" 2024-03-26T01:36:38.453 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION_ID="9" 2024-03-26T01:36:38.453 INFO:teuthology.orchestra.run.smithi069.stdout:PLATFORM_ID="platform:el9" 2024-03-26T01:36:38.454 INFO:teuthology.orchestra.run.smithi069.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T01:36:38.454 INFO:teuthology.orchestra.run.smithi069.stdout:ANSI_COLOR="0;31" 2024-03-26T01:36:38.454 INFO:teuthology.orchestra.run.smithi069.stdout:LOGO="fedora-logo-icon" 2024-03-26T01:36:38.454 INFO:teuthology.orchestra.run.smithi069.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T01:36:38.454 INFO:teuthology.orchestra.run.smithi069.stdout:HOME_URL="https://centos.org/" 2024-03-26T01:36:38.454 INFO:teuthology.orchestra.run.smithi069.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T01:36:38.454 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T01:36:38.454 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T01:36:38.454 INFO:teuthology.lock.ops:Updating smithi069.front.sepia.ceph.com on lock server 2024-03-26T01:36:38.478 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-26T01:36:38.487 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-26T01:36:38.491 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-26T01:36:38.492 DEBUG:teuthology.orchestra.run.smithi019:> test '!' -e /home/ubuntu/cephtest 2024-03-26T01:36:38.495 DEBUG:teuthology.orchestra.run.smithi069:> test '!' -e /home/ubuntu/cephtest 2024-03-26T01:36:38.510 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-26T01:36:38.516 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-26T01:36:38.516 DEBUG:teuthology.orchestra.run.smithi019:> test -z $(ls -A /var/lib/ceph) 2024-03-26T01:36:38.551 DEBUG:teuthology.orchestra.run.smithi069:> test -z $(ls -A /var/lib/ceph) 2024-03-26T01:36:38.576 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-26T01:36:38.626 INFO:teuthology.run_tasks:Running task kernel... 2024-03-26T01:36:38.642 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-26T01:36:38.642 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-26T01:36:38.642 DEBUG:teuthology.orchestra.run.smithi019:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T01:36:38.643 DEBUG:teuthology.orchestra.run.smithi069:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T01:36:38.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T01:36:38.659 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2024-03-26T01:36:38.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T01:36:38.661 DEBUG:teuthology.orchestra.run.smithi019:> uname -r 2024-03-26T01:36:38.715 INFO:teuthology.orchestra.run.smithi069.stdout:5.14.0-361.el9.x86_64 2024-03-26T01:36:38.715 INFO:teuthology.task.kernel:Running kernel on smithi069: 5.14.0-361.el9.x86_64 2024-03-26T01:36:38.715 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum install -y kernel 2024-03-26T01:36:38.719 INFO:teuthology.orchestra.run.smithi019.stdout:5.14.0-361.el9.x86_64 2024-03-26T01:36:38.720 INFO:teuthology.task.kernel:Running kernel on smithi019: 5.14.0-361.el9.x86_64 2024-03-26T01:36:38.720 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum install -y kernel 2024-03-26T01:36:40.857 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-26T01:36:41.000 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-26T01:36:42.582 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - BaseOS 3.2 MB/s | 2.0 MB 00:00 2024-03-26T01:36:42.646 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS-9 - BaseOS 4.1 MB/s | 2.0 MB 00:00 2024-03-26T01:36:43.389 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-26T01:36:43.508 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-26T01:36:45.876 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-03-26T01:36:45.892 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-03-26T01:36:49.783 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.1 MB 00:00 2024-03-26T01:36:50.328 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - CRB 7.4 MB/s | 6.1 MB 00:00 2024-03-26T01:36:51.400 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-03-26T01:36:51.694 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - Extras packages 76 kB/s | 16 kB 00:00 2024-03-26T01:36:52.460 INFO:teuthology.orchestra.run.smithi069.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-26T01:36:55.132 INFO:teuthology.orchestra.run.smithi019.stdout:Extra Packages for Enterprise Linux 6.2 MB/s | 21 MB 00:03 2024-03-26T01:36:57.451 INFO:teuthology.orchestra.run.smithi069.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-03-26T01:36:59.641 INFO:teuthology.orchestra.run.smithi069.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T01:36:59.642 INFO:teuthology.orchestra.run.smithi069.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T01:36:59.787 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-03-26T01:36:59.798 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-26T01:36:59.798 INFO:teuthology.orchestra.run.smithi069.stdout: Package Architecture Version Repository Size 2024-03-26T01:36:59.798 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-26T01:36:59.798 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-03-26T01:36:59.798 INFO:teuthology.orchestra.run.smithi069.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T01:36:59.798 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-03-26T01:36:59.798 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T01:36:59.798 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T01:36:59.798 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T01:36:59.798 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-26T01:36:59.799 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-03-26T01:36:59.799 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-03-26T01:36:59.799 INFO:teuthology.orchestra.run.smithi069.stdout:Install 4 Packages 2024-03-26T01:36:59.799 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-26T01:36:59.799 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 76 M 2024-03-26T01:36:59.799 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 123 M 2024-03-26T01:36:59.800 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-03-26T01:36:59.925 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T01:36:59.925 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T01:36:59.925 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T01:37:00.137 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T01:37:00.174 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T01:37:00.210 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T01:37:00.270 INFO:teuthology.orchestra.run.smithi019.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-26T01:37:00.394 INFO:teuthology.orchestra.run.smithi069.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 118 kB/s | 66 kB 00:00 2024-03-26T01:37:00.442 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T01:37:00.490 INFO:teuthology.orchestra.run.smithi069.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T01:37:02.436 INFO:teuthology.orchestra.run.smithi019.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T01:37:02.436 INFO:teuthology.orchestra.run.smithi019.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T01:37:02.541 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-26T01:37:02.556 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-26T01:37:02.556 INFO:teuthology.orchestra.run.smithi019.stdout: Package Architecture Version Repository Size 2024-03-26T01:37:02.556 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-26T01:37:02.556 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-26T01:37:02.557 INFO:teuthology.orchestra.run.smithi019.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T01:37:02.557 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-03-26T01:37:02.557 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T01:37:02.557 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T01:37:02.557 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T01:37:02.557 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-26T01:37:02.557 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-26T01:37:02.557 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-26T01:37:02.558 INFO:teuthology.orchestra.run.smithi019.stdout:Install 4 Packages 2024-03-26T01:37:02.558 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-26T01:37:02.558 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 76 M 2024-03-26T01:37:02.558 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 123 M 2024-03-26T01:37:02.558 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-26T01:37:02.772 INFO:teuthology.orchestra.run.smithi019.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 648 kB/s | 66 kB 00:00 2024-03-26T01:37:03.122 INFO:teuthology.orchestra.run.smithi019.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 33 MB/s | 15 MB 00:00 2024-03-26T01:37:03.531 INFO:teuthology.orchestra.run.smithi019.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 39 MB/s | 33 MB 00:00 2024-03-26T01:37:03.781 INFO:teuthology.orchestra.run.smithi019.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 27 MB/s | 27 MB 00:01 2024-03-26T01:37:03.781 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:37:03.781 INFO:teuthology.orchestra.run.smithi019.stdout:Total 62 MB/s | 76 MB 00:01 2024-03-26T01:37:04.053 INFO:teuthology.orchestra.run.smithi069.stdout:(2/4): kernel-modules-core-5.14.0-432.el9.x86_6 7.5 MB/s | 27 MB 00:03 2024-03-26T01:37:04.203 INFO:teuthology.orchestra.run.smithi069.stdout:(3/4): kernel-core-5.14.0-432.el9.x86_64.rpm 3.4 MB/s | 15 MB 00:04 2024-03-26T01:37:04.479 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-26T01:37:04.511 INFO:teuthology.orchestra.run.smithi069.stdout:(4/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 7.2 MB/s | 33 MB 00:04 2024-03-26T01:37:04.514 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:37:04.514 INFO:teuthology.orchestra.run.smithi069.stdout:Total 16 MB/s | 76 MB 00:04 2024-03-26T01:37:04.923 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-26T01:37:04.924 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-26T01:37:05.180 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-03-26T01:37:05.553 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-03-26T01:37:05.553 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-03-26T01:37:05.971 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-26T01:37:05.971 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-26T01:37:06.548 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-03-26T01:37:06.548 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-03-26T01:37:08.560 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-26T01:37:09.013 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-03-26T01:37:09.198 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T01:37:09.291 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T01:37:09.627 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T01:37:09.730 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T01:37:12.169 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T01:37:12.300 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T01:37:12.499 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T01:37:12.645 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T01:37:17.333 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T01:37:17.379 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T01:37:17.434 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:37:17.454 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:37:22.168 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:37:22.532 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:37:51.256 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:37:51.257 INFO:teuthology.orchestra.run.smithi069.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T01:37:51.257 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-26T01:37:51.300 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:37:51.640 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:37:51.640 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T01:37:51.640 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T01:37:51.640 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T01:37:52.064 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:37:52.065 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-26T01:37:52.065 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-03-26T01:37:52.065 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T01:37:52.065 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T01:37:52.065 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T01:37:52.065 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T01:37:52.065 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-26T01:37:52.065 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-03-26T01:37:52.403 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T01:37:52.513 INFO:teuthology.orchestra.run.smithi069.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T01:37:52.514 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T01:37:52.514 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T01:37:52.514 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T01:37:52.514 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T01:37:52.514 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-03-26T01:37:52.514 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi069.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T01:37:52.515 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T01:37:52.595 INFO:teuthology.orchestra.run.smithi069.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T01:37:52.595 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T01:37:52.595 DEBUG:teuthology.orchestra.run.smithi069:> sudo rpm -qi grub2-tools 2024-03-26T01:37:52.632 INFO:teuthology.orchestra.run.smithi069.stdout:Name : grub2-tools 2024-03-26T01:37:52.632 INFO:teuthology.orchestra.run.smithi069.stdout:Epoch : 1 2024-03-26T01:37:52.632 INFO:teuthology.orchestra.run.smithi069.stdout:Version : 2.06 2024-03-26T01:37:52.632 INFO:teuthology.orchestra.run.smithi069.stdout:Release : 68.el9 2024-03-26T01:37:52.632 INFO:teuthology.orchestra.run.smithi069.stdout:Architecture: x86_64 2024-03-26T01:37:52.632 INFO:teuthology.orchestra.run.smithi069.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T01:37:52.632 INFO:teuthology.orchestra.run.smithi069.stdout:Group : Unspecified 2024-03-26T01:37:52.633 INFO:teuthology.orchestra.run.smithi069.stdout:Size : 8264254 2024-03-26T01:37:52.633 INFO:teuthology.orchestra.run.smithi069.stdout:License : GPLv3+ 2024-03-26T01:37:52.633 INFO:teuthology.orchestra.run.smithi069.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T01:37:52.633 INFO:teuthology.orchestra.run.smithi069.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T01:37:52.633 INFO:teuthology.orchestra.run.smithi069.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T01:37:52.633 INFO:teuthology.orchestra.run.smithi069.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T01:37:52.633 INFO:teuthology.orchestra.run.smithi069.stdout:Packager : builder@centos.org 2024-03-26T01:37:52.633 INFO:teuthology.orchestra.run.smithi069.stdout:Vendor : CentOS 2024-03-26T01:37:52.633 INFO:teuthology.orchestra.run.smithi069.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T01:37:52.633 INFO:teuthology.orchestra.run.smithi069.stdout:Summary : Support tools for GRUB. 2024-03-26T01:37:52.633 INFO:teuthology.orchestra.run.smithi069.stdout:Description : 2024-03-26T01:37:52.633 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-26T01:37:52.633 INFO:teuthology.orchestra.run.smithi069.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T01:37:52.633 INFO:teuthology.orchestra.run.smithi069.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T01:37:52.633 INFO:teuthology.orchestra.run.smithi069.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T01:37:52.634 INFO:teuthology.orchestra.run.smithi069.stdout:hardware devices. 2024-03-26T01:37:52.634 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-26T01:37:52.634 INFO:teuthology.orchestra.run.smithi069.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T01:37:52.635 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T01:37:52.635 INFO:teuthology.task.kernel:Updating grub on smithi069 to boot 5.14.0-432.el9.x86_64 2024-03-26T01:37:52.635 DEBUG:teuthology.orchestra.run.smithi069:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T01:37:53.310 INFO:teuthology.orchestra.run.smithi069.stderr:Generating grub configuration file ... 2024-03-26T01:37:53.430 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:37:53.430 INFO:teuthology.orchestra.run.smithi019.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T01:37:53.430 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-26T01:37:53.478 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:37:53.809 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:37:53.809 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T01:37:53.810 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T01:37:53.810 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T01:37:54.222 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:37:54.222 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-26T01:37:54.222 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-26T01:37:54.222 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T01:37:54.222 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T01:37:54.223 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T01:37:54.223 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T01:37:54.223 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-26T01:37:54.223 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-26T01:37:54.537 DEBUG:teuthology.orchestra.run.smithi019:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T01:37:54.643 INFO:teuthology.orchestra.run.smithi019.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T01:37:54.644 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T01:37:54.644 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T01:37:54.644 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T01:37:54.644 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T01:37:54.644 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-03-26T01:37:54.644 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi019.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T01:37:54.644 DEBUG:teuthology.orchestra.run.smithi019:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T01:37:54.727 INFO:teuthology.orchestra.run.smithi019.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T01:37:54.728 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T01:37:54.728 DEBUG:teuthology.orchestra.run.smithi019:> sudo rpm -qi grub2-tools 2024-03-26T01:37:54.776 INFO:teuthology.orchestra.run.smithi019.stdout:Name : grub2-tools 2024-03-26T01:37:54.776 INFO:teuthology.orchestra.run.smithi019.stdout:Epoch : 1 2024-03-26T01:37:54.776 INFO:teuthology.orchestra.run.smithi019.stdout:Version : 2.06 2024-03-26T01:37:54.777 INFO:teuthology.orchestra.run.smithi019.stdout:Release : 68.el9 2024-03-26T01:37:54.777 INFO:teuthology.orchestra.run.smithi019.stdout:Architecture: x86_64 2024-03-26T01:37:54.777 INFO:teuthology.orchestra.run.smithi019.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T01:37:54.777 INFO:teuthology.orchestra.run.smithi019.stdout:Group : Unspecified 2024-03-26T01:37:54.777 INFO:teuthology.orchestra.run.smithi019.stdout:Size : 8264254 2024-03-26T01:37:54.777 INFO:teuthology.orchestra.run.smithi019.stdout:License : GPLv3+ 2024-03-26T01:37:54.777 INFO:teuthology.orchestra.run.smithi019.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T01:37:54.777 INFO:teuthology.orchestra.run.smithi019.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T01:37:54.777 INFO:teuthology.orchestra.run.smithi019.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T01:37:54.778 INFO:teuthology.orchestra.run.smithi019.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T01:37:54.778 INFO:teuthology.orchestra.run.smithi019.stdout:Packager : builder@centos.org 2024-03-26T01:37:54.778 INFO:teuthology.orchestra.run.smithi019.stdout:Vendor : CentOS 2024-03-26T01:37:54.778 INFO:teuthology.orchestra.run.smithi019.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T01:37:54.778 INFO:teuthology.orchestra.run.smithi019.stdout:Summary : Support tools for GRUB. 2024-03-26T01:37:54.778 INFO:teuthology.orchestra.run.smithi019.stdout:Description : 2024-03-26T01:37:54.778 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-26T01:37:54.778 INFO:teuthology.orchestra.run.smithi019.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T01:37:54.778 INFO:teuthology.orchestra.run.smithi019.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T01:37:54.778 INFO:teuthology.orchestra.run.smithi019.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T01:37:54.778 INFO:teuthology.orchestra.run.smithi019.stdout:hardware devices. 2024-03-26T01:37:54.778 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-26T01:37:54.779 INFO:teuthology.orchestra.run.smithi019.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T01:37:54.779 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T01:37:54.779 INFO:teuthology.task.kernel:Updating grub on smithi019 to boot 5.14.0-432.el9.x86_64 2024-03-26T01:37:54.780 DEBUG:teuthology.orchestra.run.smithi019:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T01:37:54.809 INFO:teuthology.orchestra.run.smithi069.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T01:37:54.844 INFO:teuthology.orchestra.run.smithi069.stderr:done 2024-03-26T01:37:54.846 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2024-03-26T01:37:54.861 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.gL2NXrl1hI 2024-03-26T01:37:54.861 DEBUG:teuthology.orchestra.run.smithi069:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.gL2NXrl1hI 2024-03-26T01:37:54.929 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.gL2NXrl1hI 2024-03-26T01:37:55.062 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.gL2NXrl1hI is 6KB 2024-03-26T01:37:55.112 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.gL2NXrl1hI 2024-03-26T01:37:55.126 DEBUG:teuthology.orchestra.run.smithi069:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T01:37:55.191 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T01:37:55.191 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T01:37:55.191 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T01:37:55.191 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T01:37:55.193 DEBUG:teuthology.orchestra.run.smithi069:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T01:37:55.353 DEBUG:teuthology.orchestra.run.smithi069:> sudo shutdown -r now 2024-03-26T01:37:55.363 INFO:teuthology.orchestra.run.smithi019.stderr:Generating grub configuration file ... 2024-03-26T01:37:56.877 INFO:teuthology.orchestra.run.smithi019.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T01:37:56.899 INFO:teuthology.orchestra.run.smithi019.stderr:done 2024-03-26T01:37:56.901 DEBUG:teuthology.orchestra.run.smithi019:> mktemp 2024-03-26T01:37:56.958 INFO:teuthology.orchestra.run.smithi019.stdout:/tmp/tmp.BWKGqIRL8Y 2024-03-26T01:37:56.959 DEBUG:teuthology.orchestra.run.smithi019:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.BWKGqIRL8Y 2024-03-26T01:37:57.030 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0666 /tmp/tmp.BWKGqIRL8Y 2024-03-26T01:37:57.170 DEBUG:teuthology.orchestra.remote:smithi019:/tmp/tmp.BWKGqIRL8Y is 6KB 2024-03-26T01:37:57.220 DEBUG:teuthology.orchestra.run.smithi019:> rm -fr /tmp/tmp.BWKGqIRL8Y 2024-03-26T01:37:57.235 DEBUG:teuthology.orchestra.run.smithi019:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T01:37:57.303 INFO:teuthology.orchestra.run.smithi019.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T01:37:57.303 INFO:teuthology.orchestra.run.smithi019.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T01:37:57.303 INFO:teuthology.orchestra.run.smithi019.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T01:37:57.303 INFO:teuthology.orchestra.run.smithi019.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T01:37:57.304 DEBUG:teuthology.orchestra.run.smithi019:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T01:37:57.447 DEBUG:teuthology.orchestra.run.smithi019:> sudo shutdown -r now 2024-03-26T01:38:25.382 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-03-26T01:38:25.383 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-03-26T01:38:25.384 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:38:27.452 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-03-26T01:38:27.452 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-03-26T01:38:27.452 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:38:43.875 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.69 2024-03-26T01:38:45.918 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.19 2024-03-26T01:38:52.882 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-03-26T01:38:52.883 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:38:54.921 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-03-26T01:38:54.922 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:38:57.982 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.19 2024-03-26T01:39:09.995 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-03-26T01:39:09.996 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:39:41.767 DEBUG:teuthology.orchestra.run.smithi019:> true 2024-03-26T01:39:42.293 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-03-26T01:39:42.293 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-432.el9.x86_64"... 2024-03-26T01:39:42.293 DEBUG:teuthology.orchestra.run.smithi019:> uname -r 2024-03-26T01:39:42.317 INFO:teuthology.orchestra.run.smithi019.stdout:5.14.0-432.el9.x86_64 2024-03-26T01:39:42.317 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T01:39:42.317 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T01:39:42.317 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T01:39:43.319 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-03-26T01:39:43.319 DEBUG:teuthology.orchestra.run.smithi019:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T01:39:43.383 INFO:teuthology.orchestra.run.smithi019.stdout:ttyS1 2024-03-26T01:39:43.427 DEBUG:teuthology.parallel:result is None 2024-03-26T01:39:52.886 DEBUG:teuthology.orchestra.remote:timed out 2024-03-26T01:40:04.887 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-03-26T01:40:04.888 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:40:05.247 DEBUG:teuthology.orchestra.run.smithi069:> true 2024-03-26T01:40:05.683 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-03-26T01:40:05.683 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-432.el9.x86_64"... 2024-03-26T01:40:05.683 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2024-03-26T01:40:05.699 INFO:teuthology.orchestra.run.smithi069.stdout:5.14.0-432.el9.x86_64 2024-03-26T01:40:05.699 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T01:40:05.699 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T01:40:05.699 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T01:40:06.701 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-03-26T01:40:06.701 DEBUG:teuthology.orchestra.run.smithi069:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T01:40:06.801 INFO:teuthology.orchestra.run.smithi069.stdout:ttyS1 2024-03-26T01:40:06.833 DEBUG:teuthology.parallel:result is None 2024-03-26T01:40:06.833 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-26T01:40:06.841 INFO:teuthology.task.internal:Creating test directory... 2024-03-26T01:40:06.841 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T01:40:06.844 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T01:40:06.861 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-26T01:40:06.872 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-26T01:40:06.881 INFO:teuthology.task.internal:Creating archive directory... 2024-03-26T01:40:06.882 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T01:40:06.900 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T01:40:06.932 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-26T01:40:06.941 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-26T01:40:06.941 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T01:40:06.967 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T01:40:07.001 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T01:40:07.009 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T01:40:07.015 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T01:40:07.024 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T01:40:07.026 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-26T01:40:07.033 INFO:teuthology.task.internal:Configuring sudo... 2024-03-26T01:40:07.033 DEBUG:teuthology.orchestra.run.smithi019:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T01:40:07.036 DEBUG:teuthology.orchestra.run.smithi069:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T01:40:07.086 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-26T01:40:07.097 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-26T01:40:07.097 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T01:40:07.103 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T01:40:07.141 DEBUG:teuthology.orchestra.run.smithi019:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T01:40:07.197 DEBUG:teuthology.orchestra.run.smithi019:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T01:40:07.247 DEBUG:teuthology.orchestra.run.smithi019:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T01:40:07.331 DEBUG:teuthology.orchestra.run.smithi019:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T01:40:07.371 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-26T01:40:07.371 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T01:40:07.438 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T01:40:07.480 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T01:40:07.543 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T01:40:07.584 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T01:40:07.629 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-26T01:40:07.629 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T01:40:07.697 DEBUG:teuthology.orchestra.run.smithi019:> sudo service rsyslog restart 2024-03-26T01:40:07.701 DEBUG:teuthology.orchestra.run.smithi069:> sudo service rsyslog restart 2024-03-26T01:40:07.759 INFO:teuthology.orchestra.run.smithi019.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T01:40:07.785 INFO:teuthology.orchestra.run.smithi069.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T01:40:08.048 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-26T01:40:08.056 INFO:teuthology.task.internal:Starting timer... 2024-03-26T01:40:08.056 INFO:teuthology.run_tasks:Running task pcp... 2024-03-26T01:40:08.192 INFO:teuthology.run_tasks:Running task selinux... 2024-03-26T01:40:08.202 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-26T01:40:08.203 DEBUG:teuthology.orchestra.run.smithi019:> sudo service auditd rotate 2024-03-26T01:40:08.253 INFO:teuthology.orchestra.run.smithi019.stdout:Rotating logs: 2024-03-26T01:40:08.254 DEBUG:teuthology.orchestra.run.smithi069:> sudo service auditd rotate 2024-03-26T01:40:08.313 INFO:teuthology.orchestra.run.smithi069.stdout:Rotating logs: 2024-03-26T01:40:08.315 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-26T01:40:08.315 DEBUG:teuthology.orchestra.run.smithi019:> /usr/sbin/getenforce 2024-03-26T01:40:08.350 INFO:teuthology.orchestra.run.smithi019.stdout:Permissive 2024-03-26T01:40:08.351 DEBUG:teuthology.orchestra.run.smithi069:> /usr/sbin/getenforce 2024-03-26T01:40:08.385 INFO:teuthology.orchestra.run.smithi069.stdout:Permissive 2024-03-26T01:40:08.386 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi019.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi069.front.sepia.ceph.com': 'permissive'} 2024-03-26T01:40:08.386 DEBUG:teuthology.orchestra.run.smithi019:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-26T01:40:08.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T01:40:08.419 DEBUG:teuthology.orchestra.run.smithi069:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-26T01:40:08.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T01:40:08.454 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-26T01:40:08.454 DEBUG:teuthology.orchestra.run.smithi019:> sudo /usr/sbin/setenforce permissive 2024-03-26T01:40:08.502 DEBUG:teuthology.orchestra.run.smithi069:> sudo /usr/sbin/setenforce permissive 2024-03-26T01:40:08.537 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-26T01:40:08.548 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-26T01:40:08.565 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-26T01:40:08.567 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi019.front.sepia.ceph.com,smithi069.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-26T01:44:39.446 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi019.front.sepia.ceph.com'), Remote(name='ubuntu@smithi069.front.sepia.ceph.com')] 2024-03-26T01:44:39.447 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-03-26T01:44:39.449 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:44:39.541 DEBUG:teuthology.orchestra.run.smithi019:> true 2024-03-26T01:44:39.617 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-03-26T01:44:39.617 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-03-26T01:44:39.618 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:44:39.706 DEBUG:teuthology.orchestra.run.smithi069:> true 2024-03-26T01:44:39.785 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-03-26T01:44:39.785 INFO:teuthology.run_tasks:Running task clock... 2024-03-26T01:44:39.796 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-26T01:44:39.796 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T01:44:39.796 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T01:44:39.799 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T01:44:39.799 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T01:44:39.828 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T01:44:39.844 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T01:44:39.855 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T01:44:39.872 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T01:44:39.873 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: ntpd: command not found 2024-03-26T01:44:39.887 INFO:teuthology.orchestra.run.smithi019.stdout:506 Cannot talk to daemon 2024-03-26T01:44:39.901 INFO:teuthology.orchestra.run.smithi069.stderr:sudo: ntpd: command not found 2024-03-26T01:44:39.903 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T01:44:39.915 INFO:teuthology.orchestra.run.smithi069.stdout:506 Cannot talk to daemon 2024-03-26T01:44:39.919 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T01:44:39.932 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T01:44:39.948 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T01:44:39.971 INFO:teuthology.orchestra.run.smithi019.stderr:bash: line 1: ntpq: command not found 2024-03-26T01:44:39.975 INFO:teuthology.orchestra.run.smithi019.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T01:44:39.975 INFO:teuthology.orchestra.run.smithi019.stdout:=============================================================================== 2024-03-26T01:44:39.975 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T01:44:39.975 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T01:44:39.975 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T01:44:39.975 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T01:44:40.001 INFO:teuthology.orchestra.run.smithi069.stderr:bash: line 1: ntpq: command not found 2024-03-26T01:44:40.006 INFO:teuthology.orchestra.run.smithi069.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T01:44:40.006 INFO:teuthology.orchestra.run.smithi069.stdout:=============================================================================== 2024-03-26T01:44:40.006 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T01:44:40.006 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T01:44:40.006 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T01:44:40.006 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T01:44:40.006 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-26T01:44:40.109 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a'} 2024-03-26T01:44:40.109 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-03-26T01:44:40.110 INFO:tasks.cephadm:Cluster fsid is 692607c6-eb12-11ee-95d0-87774f69a715 2024-03-26T01:44:40.110 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-26T01:44:40.110 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-03-26T01:44:40.110 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi019': '172.21.15.19', 'mon.smithi069': '172.21.15.69'} 2024-03-26T01:44:40.110 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-26T01:44:40.111 DEBUG:teuthology.orchestra.run.smithi019:> sudo hostname $(hostname -s) 2024-03-26T01:44:40.138 DEBUG:teuthology.orchestra.run.smithi069:> sudo hostname $(hostname -s) 2024-03-26T01:44:40.166 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-03-26T01:44:40.166 DEBUG:teuthology.orchestra.run.smithi019:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T01:44:40.455 INFO:teuthology.orchestra.run.smithi019.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Mar 26 01:44 /home/ubuntu/cephtest/cephadm 2024-03-26T01:44:40.455 DEBUG:teuthology.orchestra.run.smithi069:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T01:44:40.848 INFO:teuthology.orchestra.run.smithi069.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Mar 26 01:44 /home/ubuntu/cephtest/cephadm 2024-03-26T01:44:40.848 DEBUG:teuthology.orchestra.run.smithi019:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T01:44:40.865 DEBUG:teuthology.orchestra.run.smithi069:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T01:44:40.889 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-03-26T01:44:40.889 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-03-26T01:44:40.908 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-03-26T01:44:41.742 INFO:teuthology.orchestra.run.smithi069.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-26T01:44:41.793 INFO:teuthology.orchestra.run.smithi019.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-26T01:45:06.573 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T01:45:06.573 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-03-26T01:45:06.574 INFO:teuthology.orchestra.run.smithi019.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-03-26T01:45:06.574 INFO:teuthology.orchestra.run.smithi019.stdout: "repo_digests": [ 2024-03-26T01:45:06.574 INFO:teuthology.orchestra.run.smithi019.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-03-26T01:45:06.574 INFO:teuthology.orchestra.run.smithi019.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-03-26T01:45:06.574 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-03-26T01:45:06.574 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T01:45:08.151 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-03-26T01:45:08.151 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-03-26T01:45:08.151 INFO:teuthology.orchestra.run.smithi069.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-03-26T01:45:08.151 INFO:teuthology.orchestra.run.smithi069.stdout: "repo_digests": [ 2024-03-26T01:45:08.151 INFO:teuthology.orchestra.run.smithi069.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-03-26T01:45:08.152 INFO:teuthology.orchestra.run.smithi069.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-03-26T01:45:08.152 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-03-26T01:45:08.152 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-03-26T01:45:08.171 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /etc/ceph 2024-03-26T01:45:08.206 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph 2024-03-26T01:45:08.236 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 777 /etc/ceph 2024-03-26T01:45:08.275 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 777 /etc/ceph 2024-03-26T01:45:08.303 INFO:tasks.cephadm:Writing seed config... 2024-03-26T01:45:08.304 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-26T01:45:08.304 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-26T01:45:08.304 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-26T01:45:08.304 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-26T01:45:08.304 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-26T01:45:08.304 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-26T01:45:08.304 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-26T01:45:08.304 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-26T01:45:08.305 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-26T01:45:08.305 DEBUG:teuthology.orchestra.run.smithi019:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-26T01:45:08.332 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 692607c6-eb12-11ee-95d0-87774f69a715 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-26T01:45:08.332 DEBUG:teuthology.orchestra.run.smithi019:mon.smithi019> sudo journalctl -f -n 0 -u ceph-692607c6-eb12-11ee-95d0-87774f69a715@mon.smithi019.service 2024-03-26T01:45:08.374 INFO:tasks.cephadm:Bootstrapping... 2024-03-26T01:45:08.374 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid 692607c6-eb12-11ee-95d0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.19 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-26T01:45:08.577 INFO:teuthology.orchestra.run.smithi019.stderr:-------------------------------------------------------------------------------- 2024-03-26T01:45:08.577 INFO:teuthology.orchestra.run.smithi019.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', '692607c6-eb12-11ee-95d0-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.19', '--skip-admin-label'] 2024-03-26T01:45:08.601 INFO:teuthology.orchestra.run.smithi019.stderr:/bin/podman: 4.9.4-dev 2024-03-26T01:45:08.603 INFO:teuthology.orchestra.run.smithi019.stderr:Verifying podman|docker is present... 2024-03-26T01:45:08.624 INFO:teuthology.orchestra.run.smithi019.stderr:/bin/podman: 4.9.4-dev 2024-03-26T01:45:08.626 INFO:teuthology.orchestra.run.smithi019.stderr:Verifying lvm2 is present... 2024-03-26T01:45:08.627 INFO:teuthology.orchestra.run.smithi019.stderr:Verifying time synchronization is in place... 2024-03-26T01:45:08.633 INFO:teuthology.orchestra.run.smithi019.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T01:45:08.639 INFO:teuthology.orchestra.run.smithi019.stderr:systemctl: inactive 2024-03-26T01:45:08.646 INFO:teuthology.orchestra.run.smithi019.stderr:systemctl: enabled 2024-03-26T01:45:08.653 INFO:teuthology.orchestra.run.smithi019.stderr:systemctl: active 2024-03-26T01:45:08.653 INFO:teuthology.orchestra.run.smithi019.stderr:Unit chronyd.service is enabled and running 2024-03-26T01:45:08.653 INFO:teuthology.orchestra.run.smithi019.stderr:Repeating the final host check... 2024-03-26T01:45:08.673 INFO:teuthology.orchestra.run.smithi019.stderr:/bin/podman: 4.9.4-dev 2024-03-26T01:45:08.675 INFO:teuthology.orchestra.run.smithi019.stderr:podman (/bin/podman) version 4.9.4 is present 2024-03-26T01:45:08.675 INFO:teuthology.orchestra.run.smithi019.stderr:systemctl is present 2024-03-26T01:45:08.675 INFO:teuthology.orchestra.run.smithi019.stderr:lvcreate is present 2024-03-26T01:45:08.681 INFO:teuthology.orchestra.run.smithi019.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T01:45:08.687 INFO:teuthology.orchestra.run.smithi019.stderr:systemctl: inactive 2024-03-26T01:45:08.693 INFO:teuthology.orchestra.run.smithi019.stderr:systemctl: enabled 2024-03-26T01:45:08.699 INFO:teuthology.orchestra.run.smithi019.stderr:systemctl: active 2024-03-26T01:45:08.700 INFO:teuthology.orchestra.run.smithi019.stderr:Unit chronyd.service is enabled and running 2024-03-26T01:45:08.700 INFO:teuthology.orchestra.run.smithi019.stderr:Host looks OK 2024-03-26T01:45:08.700 INFO:teuthology.orchestra.run.smithi019.stderr:Cluster fsid: 692607c6-eb12-11ee-95d0-87774f69a715 2024-03-26T01:45:08.700 INFO:teuthology.orchestra.run.smithi019.stderr:Acquiring lock 139802445368720 on /run/cephadm/692607c6-eb12-11ee-95d0-87774f69a715.lock 2024-03-26T01:45:08.701 INFO:teuthology.orchestra.run.smithi019.stderr:Lock 139802445368720 acquired on /run/cephadm/692607c6-eb12-11ee-95d0-87774f69a715.lock 2024-03-26T01:45:08.701 INFO:teuthology.orchestra.run.smithi019.stderr:Verifying IP 172.21.15.19 port 3300 ... 2024-03-26T01:45:08.701 INFO:teuthology.orchestra.run.smithi019.stderr:Verifying IP 172.21.15.19 port 6789 ... 2024-03-26T01:45:08.701 INFO:teuthology.orchestra.run.smithi019.stderr:Base mon IP is 172.21.15.19, final addrv is [v2:172.21.15.19:3300,v1:172.21.15.19:6789] 2024-03-26T01:45:08.703 INFO:teuthology.orchestra.run.smithi019.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.19 metric 100 2024-03-26T01:45:08.703 INFO:teuthology.orchestra.run.smithi019.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.19 metric 100 2024-03-26T01:45:08.706 INFO:teuthology.orchestra.run.smithi019.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-03-26T01:45:08.706 INFO:teuthology.orchestra.run.smithi019.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-26T01:45:08.706 INFO:teuthology.orchestra.run.smithi019.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-26T01:45:08.709 INFO:teuthology.orchestra.run.smithi019.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-26T01:45:08.709 INFO:teuthology.orchestra.run.smithi019.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-03-26T01:45:08.709 INFO:teuthology.orchestra.run.smithi019.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-03-26T01:45:08.709 INFO:teuthology.orchestra.run.smithi019.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-26T01:45:08.709 INFO:teuthology.orchestra.run.smithi019.stderr:/sbin/ip: inet6 fe80::ec4:7aff:febd:161e/64 scope link noprefixroute 2024-03-26T01:45:08.709 INFO:teuthology.orchestra.run.smithi019.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-03-26T01:45:08.710 INFO:teuthology.orchestra.run.smithi019.stderr:Mon IP `172.21.15.19` is in CIDR network `172.21.0.0/20` 2024-03-26T01:45:08.710 INFO:teuthology.orchestra.run.smithi019.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-26T01:45:08.710 INFO:teuthology.orchestra.run.smithi019.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-26T01:45:08.816 INFO:teuthology.orchestra.run.smithi019.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-03-26T01:45:09.305 INFO:teuthology.orchestra.run.smithi019.stderr:/bin/podman: Getting image source signatures 2024-03-26T01:45:09.305 INFO:teuthology.orchestra.run.smithi019.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-03-26T01:45:09.306 INFO:teuthology.orchestra.run.smithi019.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-03-26T01:45:09.306 INFO:teuthology.orchestra.run.smithi019.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-03-26T01:45:09.306 INFO:teuthology.orchestra.run.smithi019.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-03-26T01:45:09.306 INFO:teuthology.orchestra.run.smithi019.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-03-26T01:45:09.306 INFO:teuthology.orchestra.run.smithi019.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-03-26T01:45:09.434 INFO:teuthology.orchestra.run.smithi019.stderr:/bin/podman: Writing manifest to image destination 2024-03-26T01:45:09.537 INFO:teuthology.orchestra.run.smithi019.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-03-26T01:45:09.932 INFO:teuthology.orchestra.run.smithi019.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-03-26T01:45:10.435 INFO:teuthology.orchestra.run.smithi019.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-03-26T01:45:10.436 INFO:teuthology.orchestra.run.smithi019.stderr:Extracting ceph user uid/gid from container image... 2024-03-26T01:45:10.810 INFO:teuthology.orchestra.run.smithi019.stderr:stat: 167 167 2024-03-26T01:45:11.300 INFO:teuthology.orchestra.run.smithi019.stderr:Creating initial keys... 2024-03-26T01:45:11.732 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph-authtool: AQCnKAJmqIuaJhAAS3e3Sb0ndLQx7Ut8N/AfFw== 2024-03-26T01:45:16.077 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph-authtool: AQCqKAJm3wTgMhAAEHF6CUYMXOiL2bYrdG2fww== 2024-03-26T01:45:23.171 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph-authtool: AQCzKAJmdmdABRAAV0HJQggi0zC1bvnZ/LaecQ== 2024-03-26T01:45:23.761 INFO:teuthology.orchestra.run.smithi019.stderr:Creating initial monmap... 2024-03-26T01:45:24.193 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T01:45:24.194 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-03-26T01:45:24.194 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 692607c6-eb12-11ee-95d0-87774f69a715 2024-03-26T01:45:24.194 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T01:45:24.583 INFO:teuthology.orchestra.run.smithi019.stderr:monmaptool for smithi019 [v2:172.21.15.19:3300,v1:172.21.15.19:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T01:45:24.583 INFO:teuthology.orchestra.run.smithi019.stderr:setting min_mon_release = octopus 2024-03-26T01:45:24.583 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/monmaptool: set fsid to 692607c6-eb12-11ee-95d0-87774f69a715 2024-03-26T01:45:24.583 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T01:45:24.583 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-26T01:45:24.583 INFO:teuthology.orchestra.run.smithi019.stderr:Creating mon... 2024-03-26T01:45:25.531 INFO:teuthology.orchestra.run.smithi019.stderr:create mon.smithi019 on 2024-03-26T01:45:25.742 INFO:teuthology.orchestra.run.smithi019.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-26T01:45:25.885 INFO:teuthology.orchestra.run.smithi019.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-692607c6-eb12-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-692607c6-eb12-11ee-95d0-87774f69a715.target. 2024-03-26T01:45:25.885 INFO:teuthology.orchestra.run.smithi019.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-692607c6-eb12-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-692607c6-eb12-11ee-95d0-87774f69a715.target. 2024-03-26T01:45:26.214 INFO:teuthology.orchestra.run.smithi019.stderr:systemctl: Failed to reset failed state of unit ceph-692607c6-eb12-11ee-95d0-87774f69a715@mon.smithi019.service: Unit ceph-692607c6-eb12-11ee-95d0-87774f69a715@mon.smithi019.service not loaded. 2024-03-26T01:45:26.221 INFO:teuthology.orchestra.run.smithi019.stderr:systemctl: Created symlink /etc/systemd/system/ceph-692607c6-eb12-11ee-95d0-87774f69a715.target.wants/ceph-692607c6-eb12-11ee-95d0-87774f69a715@mon.smithi019.service → /etc/systemd/system/ceph-692607c6-eb12-11ee-95d0-87774f69a715@.service. 2024-03-26T01:45:26.871 INFO:teuthology.orchestra.run.smithi019.stderr:systemctl: disabled 2024-03-26T01:45:26.877 INFO:teuthology.orchestra.run.smithi019.stderr:systemctl: inactive 2024-03-26T01:45:26.878 INFO:teuthology.orchestra.run.smithi019.stderr:firewalld.service is not enabled 2024-03-26T01:45:26.878 INFO:teuthology.orchestra.run.smithi019.stderr:Not possible to enable service . firewalld.service is not available 2024-03-26T01:45:26.878 INFO:teuthology.orchestra.run.smithi019.stderr:Waiting for mon to start... 2024-03-26T01:45:26.878 INFO:teuthology.orchestra.run.smithi019.stderr:Waiting for mon... 2024-03-26T01:45:27.666 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: cluster: 2024-03-26T01:45:27.666 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: id: 692607c6-eb12-11ee-95d0-87774f69a715 2024-03-26T01:45:27.666 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-03-26T01:45:27.666 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: 2024-03-26T01:45:27.667 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: services: 2024-03-26T01:45:27.667 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi019 (age 0.843538s) 2024-03-26T01:45:27.667 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: mgr: no daemons active 2024-03-26T01:45:27.667 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-03-26T01:45:27.667 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: 2024-03-26T01:45:27.667 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: data: 2024-03-26T01:45:27.667 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-03-26T01:45:27.667 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-03-26T01:45:27.667 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-03-26T01:45:27.667 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: pgs: 2024-03-26T01:45:27.667 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: 2024-03-26T01:45:28.694 INFO:teuthology.orchestra.run.smithi019.stderr:mon is available 2024-03-26T01:45:28.694 INFO:teuthology.orchestra.run.smithi019.stderr:Assimilating anything we can from ceph.conf... 2024-03-26T01:45:29.775 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: 2024-03-26T01:45:29.775 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: [global] 2024-03-26T01:45:29.775 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: fsid = 692607c6-eb12-11ee-95d0-87774f69a715 2024-03-26T01:45:29.775 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.19:3300,v1:172.21.15.19:6789] 2024-03-26T01:45:29.775 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-03-26T01:45:29.775 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-03-26T01:45:29.775 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-03-26T01:45:29.775 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-03-26T01:45:29.775 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: 2024-03-26T01:45:29.775 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: [mgr] 2024-03-26T01:45:29.775 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-03-26T01:45:29.776 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: 2024-03-26T01:45:29.776 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: [osd] 2024-03-26T01:45:29.776 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-03-26T01:45:29.776 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-03-26T01:45:30.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:29 smithi019 ceph-mon[24468]: from='client.? 172.21.15.19:0/200058186' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-26T01:45:30.364 INFO:teuthology.orchestra.run.smithi019.stderr:Generating new minimal ceph.conf... 2024-03-26T01:45:31.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:30 smithi019 ceph-mon[24468]: from='client.? 172.21.15.19:0/410548643' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:45:31.336 INFO:teuthology.orchestra.run.smithi019.stderr:Restarting the monitor... 2024-03-26T01:45:31.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:31 smithi019 systemd[1]: Stopping Ceph mon.smithi019 for 692607c6-eb12-11ee-95d0-87774f69a715... 2024-03-26T01:45:31.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:31 smithi019 bash[24688]: Error: no container with name or ID "ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon.smithi019" found: no such container 2024-03-26T01:45:31.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:31 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi019[24464]: 2024-03-26T01:45:31.487+0000 7f6ca3cd2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi019 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T01:45:31.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:31 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi019[24464]: 2024-03-26T01:45:31.487+0000 7f6ca3cd2700 -1 mon.smithi019@0(leader) e1 *** Got Signal Terminated *** 2024-03-26T01:45:31.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:31 smithi019 podman[24699]: 2024-03-26 01:45:31.518354282 +0000 UTC m=+0.124815977 container died 8b2b52330d88230ca3e2dc723d823ac939a6f0d9735951b04a2fca0f31250daf (image=quay.io/ceph/ceph:v17.2.0, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi019, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, release=754, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, version=8, name=centos-stream, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, ceph=True, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container) 2024-03-26T01:45:32.137 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:31 smithi019 podman[24699]: 2024-03-26 01:45:31.879030836 +0000 UTC m=+0.485492528 container cleanup 8b2b52330d88230ca3e2dc723d823ac939a6f0d9735951b04a2fca0f31250daf (image=quay.io/ceph/ceph:v17.2.0, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi019, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , architecture=x86_64, GIT_BRANCH=HEAD, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, release=754, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, com.redhat.component=centos-stream-container, distribution-scope=public, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-26T01:45:32.137 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:31 smithi019 bash[24699]: ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi019 2024-03-26T01:45:32.138 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:31 smithi019 bash[24726]: Error: no container with name or ID "ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon.smithi019" found: no such container 2024-03-26T01:45:32.389 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 podman[24714]: 2024-03-26 01:45:32.138177455 +0000 UTC m=+0.623350681 container remove 8b2b52330d88230ca3e2dc723d823ac939a6f0d9735951b04a2fca0f31250daf (image=quay.io/ceph/ceph:v17.2.0, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi019, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, version=8, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, vcs-type=git, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, RELEASE=HEAD, release=754, ceph=True) 2024-03-26T01:45:32.389 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 systemd[1]: ceph-692607c6-eb12-11ee-95d0-87774f69a715@mon.smithi019.service: Deactivated successfully. 2024-03-26T01:45:32.389 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 systemd[1]: Stopped Ceph mon.smithi019 for 692607c6-eb12-11ee-95d0-87774f69a715. 2024-03-26T01:45:32.389 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 systemd[1]: Starting Ceph mon.smithi019 for 692607c6-eb12-11ee-95d0-87774f69a715... 2024-03-26T01:45:32.389 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 podman[24797]: 2024-03-26 01:45:32.271914407 +0000 UTC m=+0.018020258 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-03-26T01:45:32.389 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 podman[24797]: 2024-03-26 01:45:32.388965756 +0000 UTC m=+0.135071632 container create b6908b497829a612ce36d3e1ff11366db26f10ad18ba6ea7804d82065d243974 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi019, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, com.redhat.component=centos-stream-container, ceph=True, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, name=centos-stream, release=754, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream) 2024-03-26T01:45:32.655 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 podman[24797]: 2024-03-26 01:45:32.632824296 +0000 UTC m=+0.378930132 container init b6908b497829a612ce36d3e1ff11366db26f10ad18ba6ea7804d82065d243974 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi019, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, io.buildah.version=1.19.8, vcs-type=git, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, version=8, RELEASE=HEAD, GIT_CLEAN=True, architecture=x86_64, GIT_BRANCH=HEAD, io.openshift.expose-services=) 2024-03-26T01:45:32.655 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 podman[24797]: 2024-03-26 01:45:32.635730295 +0000 UTC m=+0.381836120 container start b6908b497829a612ce36d3e1ff11366db26f10ad18ba6ea7804d82065d243974 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi019, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, distribution-scope=public, architecture=x86_64, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, name=centos-stream, GIT_BRANCH=HEAD, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, ceph=True, GIT_CLEAN=True, io.buildah.version=1.19.8, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-03-26T01:45:32.735 INFO:teuthology.orchestra.run.smithi019.stderr:Setting mon public_network to 172.21.0.0/20 2024-03-26T01:45:33.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T01:45:33.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-03-26T01:45:33.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: pidfile_write: ignore empty --pid-file 2024-03-26T01:45:33.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: load: jerasure load: lrc 2024-03-26T01:45:33.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: RocksDB version: 6.15.5 2024-03-26T01:45:33.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-03-26T01:45:33.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Compile date Apr 18 2022 2024-03-26T01:45:33.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: DB SUMMARY 2024-03-26T01:45:33.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: DB Session ID: 9DCFPRG67PNG8YFMZ9GO 2024-03-26T01:45:33.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: CURRENT file: CURRENT 2024-03-26T01:45:33.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T01:45:33.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-03-26T01:45:33.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi019/store.db dir, Total Num: 1, files: 000008.sst 2024-03-26T01:45:33.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi019/store.db: 000010.log size: 82703 ; 2024-03-26T01:45:33.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.error_if_exists: 0 2024-03-26T01:45:33.017 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.create_if_missing: 0 2024-03-26T01:45:33.017 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.paranoid_checks: 1 2024-03-26T01:45:33.017 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T01:45:33.017 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.env: 0x55e41dcc6860 2024-03-26T01:45:33.017 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.fs: Posix File System 2024-03-26T01:45:33.017 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.info_log: 0x55e4200c7fa0 2024-03-26T01:45:33.017 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T01:45:33.017 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.statistics: (nil) 2024-03-26T01:45:33.017 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.use_fsync: 0 2024-03-26T01:45:33.017 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.max_log_file_size: 0 2024-03-26T01:45:33.017 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T01:45:33.017 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T01:45:33.017 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T01:45:33.017 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T01:45:33.018 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.allow_fallocate: 1 2024-03-26T01:45:33.018 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T01:45:33.018 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T01:45:33.018 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.use_direct_reads: 0 2024-03-26T01:45:33.018 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T01:45:33.018 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T01:45:33.018 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.db_log_dir: 2024-03-26T01:45:33.018 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi019/store.db 2024-03-26T01:45:33.018 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T01:45:33.018 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T01:45:33.018 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T01:45:33.018 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T01:45:33.018 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T01:45:33.018 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T01:45:33.019 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T01:45:33.019 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T01:45:33.019 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.write_buffer_manager: 0x55e4201b8330 2024-03-26T01:45:33.019 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T01:45:33.019 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-03-26T01:45:33.019 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T01:45:33.019 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T01:45:33.019 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T01:45:33.019 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T01:45:33.019 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T01:45:33.019 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T01:45:33.019 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T01:45:33.019 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.unordered_write: 0 2024-03-26T01:45:33.019 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T01:45:33.019 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T01:45:33.020 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T01:45:33.020 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T01:45:33.020 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.row_cache: None 2024-03-26T01:45:33.020 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.wal_filter: None 2024-03-26T01:45:33.020 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T01:45:33.020 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T01:45:33.020 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.preserve_deletes: 0 2024-03-26T01:45:33.020 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.two_write_queues: 0 2024-03-26T01:45:33.020 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T01:45:33.020 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.atomic_flush: 0 2024-03-26T01:45:33.020 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T01:45:33.020 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T01:45:33.020 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T01:45:33.020 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.log_readahead_size: 0 2024-03-26T01:45:33.021 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T01:45:33.021 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T01:45:33.021 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T01:45:33.021 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T01:45:33.021 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T01:45:33.021 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T01:45:33.021 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.max_background_jobs: 2 2024-03-26T01:45:33.022 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.max_background_compactions: -1 2024-03-26T01:45:33.022 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.max_subcompactions: 1 2024-03-26T01:45:33.022 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T01:45:33.022 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T01:45:33.022 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T01:45:33.022 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T01:45:33.023 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T01:45:33.023 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T01:45:33.023 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T01:45:33.023 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T01:45:33.023 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.max_open_files: -1 2024-03-26T01:45:33.023 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T01:45:33.023 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T01:45:33.023 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T01:45:33.023 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T01:45:33.023 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.max_background_flushes: -1 2024-03-26T01:45:33.023 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Compression algorithms supported: 2024-03-26T01:45:33.023 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T01:45:33.023 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: kZSTD supported: 0 2024-03-26T01:45:33.024 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: kXpressCompression supported: 0 2024-03-26T01:45:33.024 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T01:45:33.024 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: kLZ4Compression supported: 1 2024-03-26T01:45:33.024 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: kBZip2Compression supported: 0 2024-03-26T01:45:33.024 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: kZlibCompression supported: 1 2024-03-26T01:45:33.024 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: kSnappyCompression supported: 1 2024-03-26T01:45:33.024 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T01:45:33.024 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi019/store.db/MANIFEST-000009 2024-03-26T01:45:33.024 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-03-26T01:45:33.024 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T01:45:33.024 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.merge_operator: 2024-03-26T01:45:33.024 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.compaction_filter: None 2024-03-26T01:45:33.024 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T01:45:33.024 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T01:45:33.025 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T01:45:33.025 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T01:45:33.025 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e420093d00) 2024-03-26T01:45:33.025 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: cache_index_and_filter_blocks: 1 2024-03-26T01:45:33.025 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T01:45:33.025 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T01:45:33.025 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: pin_top_level_index_and_filter: 1 2024-03-26T01:45:33.025 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: index_type: 0 2024-03-26T01:45:33.025 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: data_block_index_type: 0 2024-03-26T01:45:33.025 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: index_shortening: 1 2024-03-26T01:45:33.025 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T01:45:33.025 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: hash_index_allow_collision: 1 2024-03-26T01:45:33.025 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: checksum: 1 2024-03-26T01:45:33.025 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: no_block_cache: 0 2024-03-26T01:45:33.026 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: block_cache: 0x55e4200fe170 2024-03-26T01:45:33.026 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: block_cache_name: BinnedLRUCache 2024-03-26T01:45:33.026 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: block_cache_options: 2024-03-26T01:45:33.026 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: capacity : 536870912 2024-03-26T01:45:33.026 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: num_shard_bits : 4 2024-03-26T01:45:33.026 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: strict_capacity_limit : 0 2024-03-26T01:45:33.026 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: high_pri_pool_ratio: 0.000 2024-03-26T01:45:33.026 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: block_cache_compressed: (nil) 2024-03-26T01:45:33.026 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: persistent_cache: (nil) 2024-03-26T01:45:33.026 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: block_size: 4096 2024-03-26T01:45:33.026 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: block_size_deviation: 10 2024-03-26T01:45:33.026 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: block_restart_interval: 16 2024-03-26T01:45:33.026 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: index_block_restart_interval: 1 2024-03-26T01:45:33.026 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: metadata_block_size: 4096 2024-03-26T01:45:33.027 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: partition_filters: 0 2024-03-26T01:45:33.027 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: use_delta_encoding: 1 2024-03-26T01:45:33.027 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-03-26T01:45:33.027 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: whole_key_filtering: 1 2024-03-26T01:45:33.027 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: verify_compression: 0 2024-03-26T01:45:33.027 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: read_amp_bytes_per_bit: 0 2024-03-26T01:45:33.027 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: format_version: 4 2024-03-26T01:45:33.027 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: enable_index_compression: 1 2024-03-26T01:45:33.027 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: block_align: 0 2024-03-26T01:45:33.027 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T01:45:33.027 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T01:45:33.027 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.compression: NoCompression 2024-03-26T01:45:33.027 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T01:45:33.027 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T01:45:33.027 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T01:45:33.028 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.num_levels: 7 2024-03-26T01:45:33.028 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T01:45:33.028 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T01:45:33.028 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T01:45:33.028 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T01:45:33.028 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T01:45:33.028 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T01:45:33.028 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T01:45:33.028 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T01:45:33.028 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T01:45:33.028 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T01:45:33.028 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T01:45:33.028 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T01:45:33.028 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T01:45:33.028 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T01:45:33.029 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T01:45:33.029 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T01:45:33.029 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T01:45:33.029 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T01:45:33.029 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T01:45:33.029 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T01:45:33.029 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T01:45:33.029 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T01:45:33.029 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T01:45:33.029 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T01:45:33.029 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T01:45:33.029 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T01:45:33.029 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T01:45:33.029 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T01:45:33.029 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T01:45:33.030 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T01:45:33.030 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T01:45:33.030 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T01:45:33.030 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T01:45:33.030 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T01:45:33.030 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.arena_block_size: 4194304 2024-03-26T01:45:33.030 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T01:45:33.030 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T01:45:33.030 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-26T01:45:33.030 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T01:45:33.030 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T01:45:33.030 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T01:45:33.030 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T01:45:33.030 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T01:45:33.031 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T01:45:33.031 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T01:45:33.031 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T01:45:33.031 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T01:45:33.031 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T01:45:33.031 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T01:45:33.031 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.table_properties_collectors: 2024-03-26T01:45:33.031 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.inplace_update_support: 0 2024-03-26T01:45:33.031 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T01:45:33.031 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T01:45:33.031 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T01:45:33.031 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T01:45:33.031 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.bloom_locality: 0 2024-03-26T01:45:33.031 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.max_successive_merges: 0 2024-03-26T01:45:33.031 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T01:45:33.032 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T01:45:33.032 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T01:45:33.032 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T01:45:33.032 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.ttl: 2592000 2024-03-26T01:45:33.032 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T01:45:33.032 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.enable_blob_files: false 2024-03-26T01:45:33.032 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.min_blob_size: 0 2024-03-26T01:45:33.032 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T01:45:33.032 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T01:45:33.032 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T01:45:33.032 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T01:45:33.032 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi019/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-26T01:45:33.032 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-03-26T01:45:33.032 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-03-26T01:45:33.033 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711417532662834, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-03-26T01:45:33.033 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-03-26T01:45:33.033 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-26T01:45:33.033 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711417532663826, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79385, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77649, "index_size": 224, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 9884, "raw_average_key_size": 47, "raw_value_size": 71958, "raw_average_value_size": 344, "num_data_blocks": 10, "num_entries": 209, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1711417532, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "640503e3-0bd5-4503-a0b2-1c65f53f3832", "db_session_id": "9DCFPRG67PNG8YFMZ9GO"}} 2024-03-26T01:45:33.033 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-03-26T01:45:33.033 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711417532664280, "job": 1, "event": "recovery_finished"} 2024-03-26T01:45:33.033 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi019/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T01:45:33.033 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55e4200e4a80 2024-03-26T01:45:33.033 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: DB pointer 0x55e4200f4000 2024-03-26T01:45:33.033 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-03-26T01:45:33.033 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-26T01:45:33.033 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: ** DB Stats ** 2024-03-26T01:45:33.033 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T01:45:33.033 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T01:45:33.034 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T01:45:33.034 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T01:45:33.034 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T01:45:33.034 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-26T01:45:33.034 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T01:45:33.034 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: 2024-03-26T01:45:33.034 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: ** Compaction Stats [default] ** 2024-03-26T01:45:33.034 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-26T01:45:33.034 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T01:45:33.034 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: L0 2/0 79.32 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 114.2 0.00 0.00 1 0.001 0 0 2024-03-26T01:45:33.034 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Sum 2/0 79.32 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 114.2 0.00 0.00 1 0.001 0 0 2024-03-26T01:45:33.034 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 114.2 0.00 0.00 1 0.001 0 0 2024-03-26T01:45:33.034 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: 2024-03-26T01:45:33.034 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: ** Compaction Stats [default] ** 2024-03-26T01:45:33.035 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-26T01:45:33.035 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T01:45:33.035 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 114.2 0.00 0.00 1 0.001 0 0 2024-03-26T01:45:33.035 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T01:45:33.035 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T01:45:33.035 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T01:45:33.035 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T01:45:33.035 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T01:45:33.035 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T01:45:33.035 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Cumulative compaction: 0.00 GB write, 18.53 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T01:45:33.035 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Interval compaction: 0.00 GB write, 18.53 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T01:45:33.035 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T01:45:33.035 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: 2024-03-26T01:45:33.035 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T01:45:33.036 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: 2024-03-26T01:45:33.036 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: ** Compaction Stats [default] ** 2024-03-26T01:45:33.036 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-26T01:45:33.036 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T01:45:33.036 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: L0 2/0 79.32 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 114.2 0.00 0.00 1 0.001 0 0 2024-03-26T01:45:33.036 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Sum 2/0 79.32 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 114.2 0.00 0.00 1 0.001 0 0 2024-03-26T01:45:33.036 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-26T01:45:33.036 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: 2024-03-26T01:45:33.036 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: ** Compaction Stats [default] ** 2024-03-26T01:45:33.036 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-26T01:45:33.036 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T01:45:33.036 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 114.2 0.00 0.00 1 0.001 0 0 2024-03-26T01:45:33.036 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T01:45:33.036 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T01:45:33.037 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T01:45:33.037 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T01:45:33.037 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T01:45:33.037 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T01:45:33.037 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Cumulative compaction: 0.00 GB write, 18.38 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T01:45:33.037 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T01:45:33.037 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T01:45:33.037 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: 2024-03-26T01:45:33.037 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T01:45:33.037 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: starting mon.smithi019 rank 0 at public addrs [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] at bind addrs [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi019 fsid 692607c6-eb12-11ee-95d0-87774f69a715 2024-03-26T01:45:33.037 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: mon.smithi019@-1(???) e1 preinit fsid 692607c6-eb12-11ee-95d0-87774f69a715 2024-03-26T01:45:33.037 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: mon.smithi019@-1(???).mds e1 new map 2024-03-26T01:45:33.037 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: mon.smithi019@-1(???).mds e1 print_map 2024-03-26T01:45:33.038 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: e1 2024-03-26T01:45:33.038 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T01:45:33.038 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T01:45:33.038 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: legacy client fscid: -1 2024-03-26T01:45:33.038 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: 2024-03-26T01:45:33.038 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: No filesystems configured 2024-03-26T01:45:33.038 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: mon.smithi019@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T01:45:33.038 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: mon.smithi019@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T01:45:33.038 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: mon.smithi019@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T01:45:33.038 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: mon.smithi019@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T01:45:33.038 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: mon.smithi019@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-26T01:45:33.038 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: expand_channel_meta expand map: {default=false} 2024-03-26T01:45:33.038 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: expand_channel_meta from 'false' to 'false' 2024-03-26T01:45:33.038 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: expand_channel_meta expanded map: {default=false} 2024-03-26T01:45:33.039 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: expand_channel_meta expand map: {default=info} 2024-03-26T01:45:33.039 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: expand_channel_meta from 'info' to 'info' 2024-03-26T01:45:33.039 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: expand_channel_meta expanded map: {default=info} 2024-03-26T01:45:33.039 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: expand_channel_meta expand map: {default=daemon} 2024-03-26T01:45:33.039 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: expand_channel_meta from 'daemon' to 'daemon' 2024-03-26T01:45:33.039 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: expand_channel_meta expanded map: {default=daemon} 2024-03-26T01:45:33.039 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: expand_channel_meta expand map: {default=debug} 2024-03-26T01:45:33.039 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: expand_channel_meta from 'debug' to 'debug' 2024-03-26T01:45:33.039 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: expand_channel_meta expanded map: {default=debug} 2024-03-26T01:45:33.039 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 bash[24797]: b6908b497829a612ce36d3e1ff11366db26f10ad18ba6ea7804d82065d243974 2024-03-26T01:45:33.039 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 systemd[1]: Started Ceph mon.smithi019 for 692607c6-eb12-11ee-95d0-87774f69a715. 2024-03-26T01:45:33.039 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: mon.smithi019 is new leader, mons smithi019 in quorum (ranks 0) 2024-03-26T01:45:33.039 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: monmap e1: 1 mons at {smithi019=[v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0]} 2024-03-26T01:45:33.039 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: fsmap 2024-03-26T01:45:33.039 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: osdmap e1: 0 total, 0 up, 0 in 2024-03-26T01:45:33.040 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:32 smithi019 ceph-mon[24816]: mgrmap e1: no daemons active 2024-03-26T01:45:35.018 INFO:teuthology.orchestra.run.smithi019.stderr:Wrote config to /etc/ceph/ceph.conf 2024-03-26T01:45:35.018 INFO:teuthology.orchestra.run.smithi019.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-26T01:45:35.018 INFO:teuthology.orchestra.run.smithi019.stderr:Creating mgr... 2024-03-26T01:45:35.018 INFO:teuthology.orchestra.run.smithi019.stderr:Verifying port 9283 ... 2024-03-26T01:45:35.190 INFO:teuthology.orchestra.run.smithi019.stderr:systemctl: Failed to reset failed state of unit ceph-692607c6-eb12-11ee-95d0-87774f69a715@mgr.smithi019.ixuvdm.service: Unit ceph-692607c6-eb12-11ee-95d0-87774f69a715@mgr.smithi019.ixuvdm.service not loaded. 2024-03-26T01:45:35.197 INFO:teuthology.orchestra.run.smithi019.stderr:systemctl: Created symlink /etc/systemd/system/ceph-692607c6-eb12-11ee-95d0-87774f69a715.target.wants/ceph-692607c6-eb12-11ee-95d0-87774f69a715@mgr.smithi019.ixuvdm.service → /etc/systemd/system/ceph-692607c6-eb12-11ee-95d0-87774f69a715@.service. 2024-03-26T01:45:35.549 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:35 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/4107452514' entity='client.admin' 2024-03-26T01:45:35.873 INFO:teuthology.orchestra.run.smithi019.stderr:systemctl: disabled 2024-03-26T01:45:35.879 INFO:teuthology.orchestra.run.smithi019.stderr:systemctl: inactive 2024-03-26T01:45:35.880 INFO:teuthology.orchestra.run.smithi019.stderr:firewalld.service is not enabled 2024-03-26T01:45:35.880 INFO:teuthology.orchestra.run.smithi019.stderr:Not possible to enable service . firewalld.service is not available 2024-03-26T01:45:35.888 INFO:teuthology.orchestra.run.smithi019.stderr:systemctl: disabled 2024-03-26T01:45:35.894 INFO:teuthology.orchestra.run.smithi019.stderr:systemctl: inactive 2024-03-26T01:45:35.895 INFO:teuthology.orchestra.run.smithi019.stderr:firewalld.service is not enabled 2024-03-26T01:45:35.895 INFO:teuthology.orchestra.run.smithi019.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-03-26T01:45:35.896 INFO:teuthology.orchestra.run.smithi019.stderr:Waiting for mgr to start... 2024-03-26T01:45:35.896 INFO:teuthology.orchestra.run.smithi019.stderr:Waiting for mgr... 2024-03-26T01:45:36.367 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: 2024-03-26T01:45:36.367 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: { 2024-03-26T01:45:36.367 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "fsid": "692607c6-eb12-11ee-95d0-87774f69a715", 2024-03-26T01:45:36.367 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "health": { 2024-03-26T01:45:36.367 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-26T01:45:36.367 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "checks": {}, 2024-03-26T01:45:36.367 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "mutes": [] 2024-03-26T01:45:36.368 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: }, 2024-03-26T01:45:36.368 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-26T01:45:36.368 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "quorum": [ 2024-03-26T01:45:36.368 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: 0 2024-03-26T01:45:36.368 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: ], 2024-03-26T01:45:36.368 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-26T01:45:36.368 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "smithi019" 2024-03-26T01:45:36.368 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: ], 2024-03-26T01:45:36.368 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "quorum_age": 3, 2024-03-26T01:45:36.368 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "monmap": { 2024-03-26T01:45:36.368 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T01:45:36.368 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-26T01:45:36.368 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-26T01:45:36.368 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: }, 2024-03-26T01:45:36.368 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "osdmap": { 2024-03-26T01:45:36.369 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T01:45:36.369 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-26T01:45:36.369 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-26T01:45:36.369 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-26T01:45:36.369 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-26T01:45:36.369 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-26T01:45:36.369 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-26T01:45:36.369 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: }, 2024-03-26T01:45:36.369 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "pgmap": { 2024-03-26T01:45:36.369 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-26T01:45:36.369 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-26T01:45:36.369 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-26T01:45:36.369 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-26T01:45:36.369 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-26T01:45:36.370 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-26T01:45:36.370 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-26T01:45:36.370 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-26T01:45:36.370 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: }, 2024-03-26T01:45:36.370 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "fsmap": { 2024-03-26T01:45:36.370 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T01:45:36.370 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-26T01:45:36.370 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-26T01:45:36.370 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: }, 2024-03-26T01:45:36.370 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-26T01:45:36.370 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "available": false, 2024-03-26T01:45:36.370 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-26T01:45:36.370 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "modules": [ 2024-03-26T01:45:36.370 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "iostat", 2024-03-26T01:45:36.371 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "nfs", 2024-03-26T01:45:36.371 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "restful" 2024-03-26T01:45:36.371 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: ], 2024-03-26T01:45:36.371 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "services": {} 2024-03-26T01:45:36.371 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: }, 2024-03-26T01:45:36.371 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "servicemap": { 2024-03-26T01:45:36.371 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T01:45:36.371 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "modified": "2024-03-26T01:45:26.822726+0000", 2024-03-26T01:45:36.371 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "services": {} 2024-03-26T01:45:36.371 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: }, 2024-03-26T01:45:36.371 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-26T01:45:36.371 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: } 2024-03-26T01:45:36.707 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:36 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/3235463597' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T01:45:37.002 INFO:teuthology.orchestra.run.smithi019.stderr:mgr not available, waiting (1/15)... 2024-03-26T01:45:39.532 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: 2024-03-26T01:45:39.532 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: { 2024-03-26T01:45:39.532 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "fsid": "692607c6-eb12-11ee-95d0-87774f69a715", 2024-03-26T01:45:39.532 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "health": { 2024-03-26T01:45:39.532 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-26T01:45:39.532 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "checks": {}, 2024-03-26T01:45:39.532 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "mutes": [] 2024-03-26T01:45:39.532 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: }, 2024-03-26T01:45:39.532 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-26T01:45:39.533 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "quorum": [ 2024-03-26T01:45:39.533 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: 0 2024-03-26T01:45:39.533 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: ], 2024-03-26T01:45:39.533 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-26T01:45:39.533 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "smithi019" 2024-03-26T01:45:39.533 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: ], 2024-03-26T01:45:39.533 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "quorum_age": 6, 2024-03-26T01:45:39.533 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "monmap": { 2024-03-26T01:45:39.533 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T01:45:39.533 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-26T01:45:39.533 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-26T01:45:39.533 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: }, 2024-03-26T01:45:39.533 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "osdmap": { 2024-03-26T01:45:39.533 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T01:45:39.533 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-26T01:45:39.533 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-26T01:45:39.534 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-26T01:45:39.534 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-26T01:45:39.537 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-26T01:45:39.537 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-26T01:45:39.537 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: }, 2024-03-26T01:45:39.537 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "pgmap": { 2024-03-26T01:45:39.537 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-26T01:45:39.537 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-26T01:45:39.537 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-26T01:45:39.537 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-26T01:45:39.538 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-26T01:45:39.538 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-26T01:45:39.538 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-26T01:45:39.538 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-26T01:45:39.538 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: }, 2024-03-26T01:45:39.538 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "fsmap": { 2024-03-26T01:45:39.538 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T01:45:39.538 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-26T01:45:39.538 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-26T01:45:39.538 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: }, 2024-03-26T01:45:39.538 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-26T01:45:39.538 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "available": false, 2024-03-26T01:45:39.538 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-26T01:45:39.538 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "modules": [ 2024-03-26T01:45:39.538 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "iostat", 2024-03-26T01:45:39.539 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "nfs", 2024-03-26T01:45:39.539 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "restful" 2024-03-26T01:45:39.539 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: ], 2024-03-26T01:45:39.539 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "services": {} 2024-03-26T01:45:39.539 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: }, 2024-03-26T01:45:39.539 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "servicemap": { 2024-03-26T01:45:39.539 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T01:45:39.539 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "modified": "2024-03-26T01:45:26.822726+0000", 2024-03-26T01:45:39.539 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "services": {} 2024-03-26T01:45:39.539 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: }, 2024-03-26T01:45:39.539 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-26T01:45:39.539 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: } 2024-03-26T01:45:39.804 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:39 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/3507216299' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T01:45:40.092 INFO:teuthology.orchestra.run.smithi019.stderr:mgr not available, waiting (2/15)... 2024-03-26T01:45:42.219 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:41 smithi019 ceph-mon[24816]: Activating manager daemon smithi019.ixuvdm 2024-03-26T01:45:42.219 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:41 smithi019 ceph-mon[24816]: mgrmap e2: smithi019.ixuvdm(active, starting, since 0.0025925s) 2024-03-26T01:45:42.219 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:41 smithi019 ceph-mon[24816]: from='mgr.14100 172.21.15.19:0/948566091' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T01:45:42.219 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:41 smithi019 ceph-mon[24816]: from='mgr.14100 172.21.15.19:0/948566091' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T01:45:42.219 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:41 smithi019 ceph-mon[24816]: from='mgr.14100 172.21.15.19:0/948566091' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T01:45:42.219 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:41 smithi019 ceph-mon[24816]: from='mgr.14100 172.21.15.19:0/948566091' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi019"}]: dispatch 2024-03-26T01:45:42.219 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:41 smithi019 ceph-mon[24816]: from='mgr.14100 172.21.15.19:0/948566091' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mgr metadata", "who": "smithi019.ixuvdm", "id": "smithi019.ixuvdm"}]: dispatch 2024-03-26T01:45:42.220 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:41 smithi019 ceph-mon[24816]: Manager daemon smithi019.ixuvdm is now available 2024-03-26T01:45:42.220 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:41 smithi019 ceph-mon[24816]: from='mgr.14100 172.21.15.19:0/948566091' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:45:42.220 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:41 smithi019 ceph-mon[24816]: from='mgr.14100 172.21.15.19:0/948566091' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/trash_purge_schedule"}]: dispatch 2024-03-26T01:45:42.220 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:41 smithi019 ceph-mon[24816]: from='mgr.14100 172.21.15.19:0/948566091' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:45:42.220 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:41 smithi019 ceph-mon[24816]: from='mgr.14100 172.21.15.19:0/948566091' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:45:42.220 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:41 smithi019 ceph-mon[24816]: from='mgr.14100 172.21.15.19:0/948566091' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:45:42.607 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: 2024-03-26T01:45:42.607 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: { 2024-03-26T01:45:42.607 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "fsid": "692607c6-eb12-11ee-95d0-87774f69a715", 2024-03-26T01:45:42.607 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "health": { 2024-03-26T01:45:42.607 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-26T01:45:42.607 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "checks": {}, 2024-03-26T01:45:42.607 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "mutes": [] 2024-03-26T01:45:42.607 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: }, 2024-03-26T01:45:42.607 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-26T01:45:42.607 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "quorum": [ 2024-03-26T01:45:42.608 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: 0 2024-03-26T01:45:42.608 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: ], 2024-03-26T01:45:42.608 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-26T01:45:42.608 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "smithi019" 2024-03-26T01:45:42.608 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: ], 2024-03-26T01:45:42.608 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "quorum_age": 9, 2024-03-26T01:45:42.608 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "monmap": { 2024-03-26T01:45:42.608 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T01:45:42.608 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-26T01:45:42.608 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-26T01:45:42.608 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: }, 2024-03-26T01:45:42.608 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "osdmap": { 2024-03-26T01:45:42.608 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T01:45:42.608 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-26T01:45:42.608 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-26T01:45:42.609 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-26T01:45:42.612 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-26T01:45:42.612 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-26T01:45:42.612 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-26T01:45:42.612 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: }, 2024-03-26T01:45:42.612 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "pgmap": { 2024-03-26T01:45:42.612 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-26T01:45:42.612 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-26T01:45:42.612 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-26T01:45:42.613 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-26T01:45:42.613 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-26T01:45:42.613 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-26T01:45:42.613 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-26T01:45:42.613 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-26T01:45:42.613 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: }, 2024-03-26T01:45:42.613 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "fsmap": { 2024-03-26T01:45:42.613 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T01:45:42.613 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-26T01:45:42.613 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-26T01:45:42.613 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: }, 2024-03-26T01:45:42.613 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-26T01:45:42.613 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "available": false, 2024-03-26T01:45:42.613 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-26T01:45:42.613 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "modules": [ 2024-03-26T01:45:42.613 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "iostat", 2024-03-26T01:45:42.614 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "nfs", 2024-03-26T01:45:42.614 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "restful" 2024-03-26T01:45:42.614 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: ], 2024-03-26T01:45:42.614 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "services": {} 2024-03-26T01:45:42.614 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: }, 2024-03-26T01:45:42.614 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "servicemap": { 2024-03-26T01:45:42.614 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T01:45:42.614 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "modified": "2024-03-26T01:45:26.822726+0000", 2024-03-26T01:45:42.614 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "services": {} 2024-03-26T01:45:42.614 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: }, 2024-03-26T01:45:42.614 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-26T01:45:42.614 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: } 2024-03-26T01:45:43.162 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:42 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/356190805' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T01:45:43.163 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:42 smithi019 ceph-mon[24816]: mgrmap e3: smithi019.ixuvdm(active, since 1.00806s) 2024-03-26T01:45:43.165 INFO:teuthology.orchestra.run.smithi019.stderr:mgr not available, waiting (3/15)... 2024-03-26T01:45:45.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:44 smithi019 ceph-mon[24816]: mgrmap e4: smithi019.ixuvdm(active, since 2s) 2024-03-26T01:45:45.825 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: 2024-03-26T01:45:45.825 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: { 2024-03-26T01:45:45.825 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "fsid": "692607c6-eb12-11ee-95d0-87774f69a715", 2024-03-26T01:45:45.825 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "health": { 2024-03-26T01:45:45.825 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-26T01:45:45.825 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "checks": {}, 2024-03-26T01:45:45.825 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "mutes": [] 2024-03-26T01:45:45.825 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: }, 2024-03-26T01:45:45.825 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-26T01:45:45.825 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "quorum": [ 2024-03-26T01:45:45.825 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: 0 2024-03-26T01:45:45.826 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: ], 2024-03-26T01:45:45.826 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-26T01:45:45.826 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "smithi019" 2024-03-26T01:45:45.826 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: ], 2024-03-26T01:45:45.826 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "quorum_age": 13, 2024-03-26T01:45:45.826 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "monmap": { 2024-03-26T01:45:45.826 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T01:45:45.826 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-26T01:45:45.826 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-26T01:45:45.826 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: }, 2024-03-26T01:45:45.826 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "osdmap": { 2024-03-26T01:45:45.826 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T01:45:45.826 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-26T01:45:45.826 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-26T01:45:45.827 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-26T01:45:45.830 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-26T01:45:45.830 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-26T01:45:45.831 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-26T01:45:45.831 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: }, 2024-03-26T01:45:45.831 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "pgmap": { 2024-03-26T01:45:45.831 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-26T01:45:45.831 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-26T01:45:45.831 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-26T01:45:45.831 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-26T01:45:45.831 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-26T01:45:45.831 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-26T01:45:45.831 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-26T01:45:45.831 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-26T01:45:45.831 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: }, 2024-03-26T01:45:45.831 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "fsmap": { 2024-03-26T01:45:45.831 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T01:45:45.832 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-26T01:45:45.832 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-26T01:45:45.832 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: }, 2024-03-26T01:45:45.832 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-26T01:45:45.832 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "available": true, 2024-03-26T01:45:45.832 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-26T01:45:45.832 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "modules": [ 2024-03-26T01:45:45.832 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "iostat", 2024-03-26T01:45:45.832 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "nfs", 2024-03-26T01:45:45.832 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "restful" 2024-03-26T01:45:45.832 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: ], 2024-03-26T01:45:45.832 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "services": {} 2024-03-26T01:45:45.832 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: }, 2024-03-26T01:45:45.832 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "servicemap": { 2024-03-26T01:45:45.833 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T01:45:45.833 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "modified": "2024-03-26T01:45:26.822726+0000", 2024-03-26T01:45:45.833 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "services": {} 2024-03-26T01:45:45.833 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: }, 2024-03-26T01:45:45.833 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-26T01:45:45.833 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: } 2024-03-26T01:45:46.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:45 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/1843297142' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T01:45:46.387 INFO:teuthology.orchestra.run.smithi019.stderr:mgr is available 2024-03-26T01:45:47.021 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: 2024-03-26T01:45:47.021 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: [global] 2024-03-26T01:45:47.022 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: fsid = 692607c6-eb12-11ee-95d0-87774f69a715 2024-03-26T01:45:47.022 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-03-26T01:45:47.022 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-03-26T01:45:47.022 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-03-26T01:45:47.022 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-03-26T01:45:47.022 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: 2024-03-26T01:45:47.022 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: [mgr] 2024-03-26T01:45:47.022 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-03-26T01:45:47.022 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: 2024-03-26T01:45:47.022 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: [osd] 2024-03-26T01:45:47.022 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-03-26T01:45:47.022 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-03-26T01:45:47.323 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:47 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/6358248' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T01:45:47.602 INFO:teuthology.orchestra.run.smithi019.stderr:Enabling cephadm module... 2024-03-26T01:45:48.972 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:48 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/415239414' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-26T01:45:49.845 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:49 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/415239414' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-26T01:45:49.845 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:49 smithi019 ceph-mon[24816]: mgrmap e5: smithi019.ixuvdm(active, since 6s) 2024-03-26T01:45:49.846 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: { 2024-03-26T01:45:49.846 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "epoch": 5, 2024-03-26T01:45:49.846 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "available": true, 2024-03-26T01:45:49.846 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "active_name": "smithi019.ixuvdm", 2024-03-26T01:45:49.846 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_standby": 0 2024-03-26T01:45:49.846 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: } 2024-03-26T01:45:50.431 INFO:teuthology.orchestra.run.smithi019.stderr:Waiting for the mgr to restart... 2024-03-26T01:45:50.431 INFO:teuthology.orchestra.run.smithi019.stderr:Waiting for mgr epoch 5... 2024-03-26T01:45:51.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:50 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/1652865628' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T01:45:55.737 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: { 2024-03-26T01:45:55.737 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-03-26T01:45:55.737 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "initialized": true 2024-03-26T01:45:55.737 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: } 2024-03-26T01:45:56.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:55 smithi019 ceph-mon[24816]: Active manager daemon smithi019.ixuvdm restarted 2024-03-26T01:45:56.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:55 smithi019 ceph-mon[24816]: Activating manager daemon smithi019.ixuvdm 2024-03-26T01:45:56.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:55 smithi019 ceph-mon[24816]: osdmap e2: 0 total, 0 up, 0 in 2024-03-26T01:45:56.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:55 smithi019 ceph-mon[24816]: mgrmap e6: smithi019.ixuvdm(active, starting, since 0.0525483s) 2024-03-26T01:45:56.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:55 smithi019 ceph-mon[24816]: from='mgr.14120 172.21.15.19:0/2439737324' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi019"}]: dispatch 2024-03-26T01:45:56.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:55 smithi019 ceph-mon[24816]: from='mgr.14120 172.21.15.19:0/2439737324' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mgr metadata", "who": "smithi019.ixuvdm", "id": "smithi019.ixuvdm"}]: dispatch 2024-03-26T01:45:56.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:55 smithi019 ceph-mon[24816]: from='mgr.14120 172.21.15.19:0/2439737324' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T01:45:56.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:55 smithi019 ceph-mon[24816]: from='mgr.14120 172.21.15.19:0/2439737324' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T01:45:56.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:55 smithi019 ceph-mon[24816]: from='mgr.14120 172.21.15.19:0/2439737324' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T01:45:56.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:55 smithi019 ceph-mon[24816]: Manager daemon smithi019.ixuvdm is now available 2024-03-26T01:45:56.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:55 smithi019 ceph-mon[24816]: from='mgr.14120 172.21.15.19:0/2439737324' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:45:56.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:55 smithi019 ceph-mon[24816]: from='mgr.14120 172.21.15.19:0/2439737324' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:45:56.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:55 smithi019 ceph-mon[24816]: from='mgr.14120 172.21.15.19:0/2439737324' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:45:56.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:55 smithi019 ceph-mon[24816]: from='mgr.14120 172.21.15.19:0/2439737324' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:45:56.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:55 smithi019 ceph-mon[24816]: from='mgr.14120 172.21.15.19:0/2439737324' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:45:56.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:55 smithi019 ceph-mon[24816]: from='mgr.14120 172.21.15.19:0/2439737324' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:45:56.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:55 smithi019 ceph-mon[24816]: from='mgr.14120 172.21.15.19:0/2439737324' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/trash_purge_schedule"}]: dispatch 2024-03-26T01:45:56.304 INFO:teuthology.orchestra.run.smithi019.stderr:mgr epoch 5 is available 2024-03-26T01:45:56.304 INFO:teuthology.orchestra.run.smithi019.stderr:Setting orchestrator backend to cephadm... 2024-03-26T01:45:56.924 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:56 smithi019 ceph-mon[24816]: mgrmap e7: smithi019.ixuvdm(active, since 1.05526s) 2024-03-26T01:45:56.925 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:56 smithi019 ceph-mon[24816]: [26/Mar/2024:01:45:56] ENGINE Bus STARTING 2024-03-26T01:45:58.006 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:57 smithi019 ceph-mon[24816]: mgrmap e8: smithi019.ixuvdm(active, since 2s) 2024-03-26T01:45:58.006 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:57 smithi019 ceph-mon[24816]: [26/Mar/2024:01:45:56] ENGINE Serving on https://172.21.15.19:7150 2024-03-26T01:45:58.006 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:57 smithi019 ceph-mon[24816]: [26/Mar/2024:01:45:56] ENGINE Bus STARTED 2024-03-26T01:45:58.007 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:57 smithi019 ceph-mon[24816]: from='mgr.14120 172.21.15.19:0/2439737324' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:45:58.007 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:57 smithi019 ceph-mon[24816]: from='mgr.14120 172.21.15.19:0/2439737324' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:45:58.007 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:57 smithi019 ceph-mon[24816]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:45:58.007 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:57 smithi019 ceph-mon[24816]: from='mgr.14120 172.21.15.19:0/2439737324' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:45:58.007 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:57 smithi019 ceph-mon[24816]: from='mgr.14120 172.21.15.19:0/2439737324' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:45:58.008 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: value unchanged 2024-03-26T01:45:58.575 INFO:teuthology.orchestra.run.smithi019.stderr:Generating ssh key... 2024-03-26T01:45:58.924 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:45:58 smithi019 ceph-mon[24816]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:46:00.591 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:00 smithi019 ceph-mon[24816]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:46:00.591 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:00 smithi019 ceph-mon[24816]: Generating ssh key... 2024-03-26T01:46:00.591 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:00 smithi019 ceph-mon[24816]: from='mgr.14120 172.21.15.19:0/2439737324' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:46:00.591 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:00 smithi019 ceph-mon[24816]: from='mgr.14120 172.21.15.19:0/2439737324' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:46:00.591 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLMruK2D0X1xA+ITgmCDbxmPjeYnL8dMEP5mONTiKGrf/4+eY605MMeREH7CaGSXWgNHsFoHjYNwpn2NSGuD7+t7DssTEmdlbQuGmHeLQIymNUjWL9I3J8rNkzkjPo8ymuMtnx2yutEylBtlK2iyCSi7aLER5OVu6abXlHCIMrXzJ+JPeWREEpow/8H1t/eIiIkhkx2HcMvFUT9ROK/rLutmNPrgjhTUl1thq/qlQ6vF0KW7/CeCfbTgMupUKf4BU/WSgi/erTGoQ1cszFYcI1rHuRDnVpzTI5P6FpUG+1WKdhd3ujtWjrPXRueexcJYuNDEeFjwfdo/teNIBh+jtIKkEQ7pj7tSdeM8ospkI6Cf4WU3djAAhN4pHeOj/wd00Ps1Myr5Cf6UYrI45MXxIFusXQ8Wo/gNhJa+lDUwwnuN87Qy3irHUz46vGKWS2JcvYiiNiiI5B43zwBYc+zi/JXLqik2YKxRYzNgH72fOKCPIrfZFhyshdsVIbdexkZoE= ceph-692607c6-eb12-11ee-95d0-87774f69a715 2024-03-26T01:46:01.095 INFO:teuthology.orchestra.run.smithi019.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-26T01:46:01.096 INFO:teuthology.orchestra.run.smithi019.stderr:Adding key to root@localhost authorized_keys... 2024-03-26T01:46:01.096 INFO:teuthology.orchestra.run.smithi019.stderr:Adding host smithi019... 2024-03-26T01:46:01.514 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:01 smithi019 ceph-mon[24816]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:46:02.751 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: Added host 'smithi019' with addr '172.21.15.19' 2024-03-26T01:46:03.336 INFO:teuthology.orchestra.run.smithi019.stderr:Deploying mon service with default placement... 2024-03-26T01:46:03.720 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:03 smithi019 ceph-mon[24816]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi019", "addr": "172.21.15.19", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:46:03.720 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:03 smithi019 ceph-mon[24816]: Deploying cephadm binary to smithi019 2024-03-26T01:46:03.720 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:03 smithi019 ceph-mon[24816]: from='mgr.14120 172.21.15.19:0/2439737324' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:46:03.720 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:03 smithi019 ceph-mon[24816]: from='mgr.14120 172.21.15.19:0/2439737324' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:46:03.945 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: Scheduled mon update... 2024-03-26T01:46:04.736 INFO:teuthology.orchestra.run.smithi019.stderr:Deploying mgr service with default placement... 2024-03-26T01:46:04.884 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:04 smithi019 ceph-mon[24816]: Added host smithi019 2024-03-26T01:46:04.884 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:04 smithi019 ceph-mon[24816]: from='mgr.14120 172.21.15.19:0/2439737324' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:46:05.589 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-03-26T01:46:05.764 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:05 smithi019 ceph-mon[24816]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:46:05.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:05 smithi019 ceph-mon[24816]: Saving service mon spec with placement count:5 2024-03-26T01:46:05.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:05 smithi019 ceph-mon[24816]: from='mgr.14120 172.21.15.19:0/2439737324' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:46:06.329 INFO:teuthology.orchestra.run.smithi019.stderr:Deploying crash service with default placement... 2024-03-26T01:46:06.936 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:06 smithi019 ceph-mon[24816]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:46:06.936 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:06 smithi019 ceph-mon[24816]: Saving service mgr spec with placement count:2 2024-03-26T01:46:06.936 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: Scheduled crash update... 2024-03-26T01:46:07.621 INFO:teuthology.orchestra.run.smithi019.stderr:Deploying prometheus service with default placement... 2024-03-26T01:46:08.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:07 smithi019 ceph-mon[24816]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:46:08.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:07 smithi019 ceph-mon[24816]: Saving service crash spec with placement * 2024-03-26T01:46:08.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:07 smithi019 ceph-mon[24816]: from='mgr.14120 172.21.15.19:0/2439737324' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:46:08.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:07 smithi019 ceph-mon[24816]: from='mgr.14120 172.21.15.19:0/2439737324' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:46:08.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:07 smithi019 ceph-mon[24816]: from='mgr.14120 172.21.15.19:0/2439737324' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:46:08.316 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-03-26T01:46:08.955 INFO:teuthology.orchestra.run.smithi019.stderr:Deploying grafana service with default placement... 2024-03-26T01:46:09.598 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:09 smithi019 ceph-mon[24816]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:46:09.599 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:09 smithi019 ceph-mon[24816]: Saving service prometheus spec with placement count:1 2024-03-26T01:46:09.599 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:09 smithi019 ceph-mon[24816]: from='mgr.14120 172.21.15.19:0/2439737324' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:46:09.901 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-03-26T01:46:10.463 INFO:teuthology.orchestra.run.smithi019.stderr:Deploying node-exporter service with default placement... 2024-03-26T01:46:10.713 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:10 smithi019 ceph-mon[24816]: from='mgr.14120 172.21.15.19:0/2439737324' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:46:10.713 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:10 smithi019 ceph-mon[24816]: from='mgr.14120 172.21.15.19:0/2439737324' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:46:10.713 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:10 smithi019 ceph-mon[24816]: from='mgr.14120 172.21.15.19:0/2439737324' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:46:11.136 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-03-26T01:46:11.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:11 smithi019 ceph-mon[24816]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:46:11.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:11 smithi019 ceph-mon[24816]: Saving service grafana spec with placement count:1 2024-03-26T01:46:11.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:11 smithi019 ceph-mon[24816]: from='mgr.14120 172.21.15.19:0/2439737324' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:46:12.132 INFO:teuthology.orchestra.run.smithi019.stderr:Deploying alertmanager service with default placement... 2024-03-26T01:46:12.687 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:12 smithi019 ceph-mon[24816]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:46:12.687 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:12 smithi019 ceph-mon[24816]: Saving service node-exporter spec with placement * 2024-03-26T01:46:13.252 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-03-26T01:46:14.506 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:14 smithi019 ceph-mon[24816]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:46:14.506 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:14 smithi019 ceph-mon[24816]: Saving service alertmanager spec with placement count:1 2024-03-26T01:46:14.507 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:14 smithi019 ceph-mon[24816]: from='mgr.14120 172.21.15.19:0/2439737324' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:46:15.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:15 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/3897147211' entity='client.admin' 2024-03-26T01:46:16.415 INFO:teuthology.orchestra.run.smithi019.stderr:Enabling the dashboard module... 2024-03-26T01:46:16.738 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:16 smithi019 ceph-mon[24816]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:46:16.738 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:16 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/628574438' entity='client.admin' 2024-03-26T01:46:17.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:17 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/2028062282' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-26T01:46:18.725 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:18 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/2028062282' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-26T01:46:18.725 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:18 smithi019 ceph-mon[24816]: mgrmap e9: smithi019.ixuvdm(active, since 22s) 2024-03-26T01:46:18.867 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: { 2024-03-26T01:46:18.867 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "epoch": 9, 2024-03-26T01:46:18.867 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "available": true, 2024-03-26T01:46:18.867 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "active_name": "smithi019.ixuvdm", 2024-03-26T01:46:18.867 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "num_standby": 0 2024-03-26T01:46:18.868 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: } 2024-03-26T01:46:19.436 INFO:teuthology.orchestra.run.smithi019.stderr:Waiting for the mgr to restart... 2024-03-26T01:46:19.436 INFO:teuthology.orchestra.run.smithi019.stderr:Waiting for mgr epoch 9... 2024-03-26T01:46:19.593 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:19 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/2806764367' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T01:46:23.942 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:23 smithi019 ceph-mon[24816]: Active manager daemon smithi019.ixuvdm restarted 2024-03-26T01:46:23.942 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:23 smithi019 ceph-mon[24816]: Activating manager daemon smithi019.ixuvdm 2024-03-26T01:46:23.942 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:23 smithi019 ceph-mon[24816]: osdmap e3: 0 total, 0 up, 0 in 2024-03-26T01:46:24.702 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: { 2024-03-26T01:46:24.702 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-03-26T01:46:24.702 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: "initialized": true 2024-03-26T01:46:24.702 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: } 2024-03-26T01:46:25.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:24 smithi019 ceph-mon[24816]: mgrmap e10: smithi019.ixuvdm(active, starting, since 0.187327s) 2024-03-26T01:46:25.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:24 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi019"}]: dispatch 2024-03-26T01:46:25.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:24 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mgr metadata", "who": "smithi019.ixuvdm", "id": "smithi019.ixuvdm"}]: dispatch 2024-03-26T01:46:25.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:24 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T01:46:25.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:24 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T01:46:25.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:24 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T01:46:25.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:24 smithi019 ceph-mon[24816]: Manager daemon smithi019.ixuvdm is now available 2024-03-26T01:46:25.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:24 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:46:25.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:24 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:46:25.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:24 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:46:25.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:24 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/trash_purge_schedule"}]: dispatch 2024-03-26T01:46:25.480 INFO:teuthology.orchestra.run.smithi019.stderr:mgr epoch 9 is available 2024-03-26T01:46:25.480 INFO:teuthology.orchestra.run.smithi019.stderr:Generating a dashboard self-signed certificate... 2024-03-26T01:46:25.756 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:25 smithi019 ceph-mon[24816]: mgrmap e11: smithi019.ixuvdm(active, since 1.19416s) 2024-03-26T01:46:25.757 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:25 smithi019 ceph-mon[24816]: [26/Mar/2024:01:46:24] ENGINE Bus STARTING 2024-03-26T01:46:25.757 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:25 smithi019 ceph-mon[24816]: [26/Mar/2024:01:46:24] ENGINE Serving on https://172.21.15.19:7150 2024-03-26T01:46:25.757 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:25 smithi019 ceph-mon[24816]: [26/Mar/2024:01:46:24] ENGINE Bus STARTED 2024-03-26T01:46:25.757 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:25 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:46:25.757 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:25 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:46:26.267 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: Self-signed certificate created 2024-03-26T01:46:26.910 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:26 smithi019 ceph-mon[24816]: mgrmap e12: smithi019.ixuvdm(active, since 2s) 2024-03-26T01:46:26.910 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:26 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:46:26.910 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:26 smithi019 ceph-mon[24816]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:46:26.910 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:26 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:46:26.910 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:26 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:46:26.912 INFO:teuthology.orchestra.run.smithi019.stderr:Creating initial admin user... 2024-03-26T01:46:28.028 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$vS2OcMWp0ILE8n44/lU/Ju30glskJZACySs1HRC0QBjm6szkniCQu", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711417588, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-26T01:46:28.615 INFO:teuthology.orchestra.run.smithi019.stderr:Fetching dashboard port number... 2024-03-26T01:46:29.192 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:29 smithi019 ceph-mon[24816]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:46:29.192 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:29 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:46:29.579 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: 8443 2024-03-26T01:46:30.391 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:30 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/1003039462' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-26T01:46:30.403 INFO:teuthology.orchestra.run.smithi019.stderr:systemctl: disabled 2024-03-26T01:46:30.411 INFO:teuthology.orchestra.run.smithi019.stderr:systemctl: inactive 2024-03-26T01:46:30.411 INFO:teuthology.orchestra.run.smithi019.stderr:firewalld.service is not enabled 2024-03-26T01:46:30.411 INFO:teuthology.orchestra.run.smithi019.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-26T01:46:30.414 INFO:teuthology.orchestra.run.smithi019.stderr:Ceph Dashboard is now available at: 2024-03-26T01:46:30.414 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-26T01:46:30.414 INFO:teuthology.orchestra.run.smithi019.stderr: URL: https://smithi019.front.sepia.ceph.com:8443/ 2024-03-26T01:46:30.414 INFO:teuthology.orchestra.run.smithi019.stderr: User: admin 2024-03-26T01:46:30.414 INFO:teuthology.orchestra.run.smithi019.stderr: Password: ld7cgoykgt 2024-03-26T01:46:30.414 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-26T01:46:30.414 INFO:teuthology.orchestra.run.smithi019.stderr:Enabling autotune for osd_memory_target 2024-03-26T01:46:31.514 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:31 smithi019 ceph-mon[24816]: mgrmap e13: smithi019.ixuvdm(active, since 6s) 2024-03-26T01:46:32.675 INFO:teuthology.orchestra.run.smithi019.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-03-26T01:46:33.467 INFO:teuthology.orchestra.run.smithi019.stderr:You can access the Ceph CLI with: 2024-03-26T01:46:33.468 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-26T01:46:33.468 INFO:teuthology.orchestra.run.smithi019.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-26T01:46:33.468 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-26T01:46:33.468 INFO:teuthology.orchestra.run.smithi019.stderr:Please consider enabling telemetry to help improve Ceph: 2024-03-26T01:46:33.468 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-26T01:46:33.468 INFO:teuthology.orchestra.run.smithi019.stderr: ceph telemetry on 2024-03-26T01:46:33.468 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-26T01:46:33.468 INFO:teuthology.orchestra.run.smithi019.stderr:For more information see: 2024-03-26T01:46:33.468 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-26T01:46:33.468 INFO:teuthology.orchestra.run.smithi019.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-03-26T01:46:33.468 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-26T01:46:33.468 INFO:teuthology.orchestra.run.smithi019.stderr:Bootstrap complete. 2024-03-26T01:46:33.511 INFO:tasks.cephadm:Fetching config... 2024-03-26T01:46:33.512 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-26T01:46:33.512 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-26T01:46:33.529 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-26T01:46:33.529 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-26T01:46:33.529 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-26T01:46:33.585 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-26T01:46:33.586 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-26T01:46:33.586 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/mon.smithi019/keyring of=/dev/stdout 2024-03-26T01:46:33.656 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-26T01:46:33.656 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-26T01:46:33.656 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-26T01:46:33.712 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-26T01:46:33.712 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLMruK2D0X1xA+ITgmCDbxmPjeYnL8dMEP5mONTiKGrf/4+eY605MMeREH7CaGSXWgNHsFoHjYNwpn2NSGuD7+t7DssTEmdlbQuGmHeLQIymNUjWL9I3J8rNkzkjPo8ymuMtnx2yutEylBtlK2iyCSi7aLER5OVu6abXlHCIMrXzJ+JPeWREEpow/8H1t/eIiIkhkx2HcMvFUT9ROK/rLutmNPrgjhTUl1thq/qlQ6vF0KW7/CeCfbTgMupUKf4BU/WSgi/erTGoQ1cszFYcI1rHuRDnVpzTI5P6FpUG+1WKdhd3ujtWjrPXRueexcJYuNDEeFjwfdo/teNIBh+jtIKkEQ7pj7tSdeM8ospkI6Cf4WU3djAAhN4pHeOj/wd00Ps1Myr5Cf6UYrI45MXxIFusXQ8Wo/gNhJa+lDUwwnuN87Qy3irHUz46vGKWS2JcvYiiNiiI5B43zwBYc+zi/JXLqik2YKxRYzNgH72fOKCPIrfZFhyshdsVIbdexkZoE= ceph-692607c6-eb12-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T01:46:33.759 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:33 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/2957923294' entity='client.admin' 2024-03-26T01:46:33.794 INFO:teuthology.orchestra.run.smithi019.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLMruK2D0X1xA+ITgmCDbxmPjeYnL8dMEP5mONTiKGrf/4+eY605MMeREH7CaGSXWgNHsFoHjYNwpn2NSGuD7+t7DssTEmdlbQuGmHeLQIymNUjWL9I3J8rNkzkjPo8ymuMtnx2yutEylBtlK2iyCSi7aLER5OVu6abXlHCIMrXzJ+JPeWREEpow/8H1t/eIiIkhkx2HcMvFUT9ROK/rLutmNPrgjhTUl1thq/qlQ6vF0KW7/CeCfbTgMupUKf4BU/WSgi/erTGoQ1cszFYcI1rHuRDnVpzTI5P6FpUG+1WKdhd3ujtWjrPXRueexcJYuNDEeFjwfdo/teNIBh+jtIKkEQ7pj7tSdeM8ospkI6Cf4WU3djAAhN4pHeOj/wd00Ps1Myr5Cf6UYrI45MXxIFusXQ8Wo/gNhJa+lDUwwnuN87Qy3irHUz46vGKWS2JcvYiiNiiI5B43zwBYc+zi/JXLqik2YKxRYzNgH72fOKCPIrfZFhyshdsVIbdexkZoE= ceph-692607c6-eb12-11ee-95d0-87774f69a715 2024-03-26T01:46:33.808 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLMruK2D0X1xA+ITgmCDbxmPjeYnL8dMEP5mONTiKGrf/4+eY605MMeREH7CaGSXWgNHsFoHjYNwpn2NSGuD7+t7DssTEmdlbQuGmHeLQIymNUjWL9I3J8rNkzkjPo8ymuMtnx2yutEylBtlK2iyCSi7aLER5OVu6abXlHCIMrXzJ+JPeWREEpow/8H1t/eIiIkhkx2HcMvFUT9ROK/rLutmNPrgjhTUl1thq/qlQ6vF0KW7/CeCfbTgMupUKf4BU/WSgi/erTGoQ1cszFYcI1rHuRDnVpzTI5P6FpUG+1WKdhd3ujtWjrPXRueexcJYuNDEeFjwfdo/teNIBh+jtIKkEQ7pj7tSdeM8ospkI6Cf4WU3djAAhN4pHeOj/wd00Ps1Myr5Cf6UYrI45MXxIFusXQ8Wo/gNhJa+lDUwwnuN87Qy3irHUz46vGKWS2JcvYiiNiiI5B43zwBYc+zi/JXLqik2YKxRYzNgH72fOKCPIrfZFhyshdsVIbdexkZoE= ceph-692607c6-eb12-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T01:46:33.853 INFO:teuthology.orchestra.run.smithi069.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLMruK2D0X1xA+ITgmCDbxmPjeYnL8dMEP5mONTiKGrf/4+eY605MMeREH7CaGSXWgNHsFoHjYNwpn2NSGuD7+t7DssTEmdlbQuGmHeLQIymNUjWL9I3J8rNkzkjPo8ymuMtnx2yutEylBtlK2iyCSi7aLER5OVu6abXlHCIMrXzJ+JPeWREEpow/8H1t/eIiIkhkx2HcMvFUT9ROK/rLutmNPrgjhTUl1thq/qlQ6vF0KW7/CeCfbTgMupUKf4BU/WSgi/erTGoQ1cszFYcI1rHuRDnVpzTI5P6FpUG+1WKdhd3ujtWjrPXRueexcJYuNDEeFjwfdo/teNIBh+jtIKkEQ7pj7tSdeM8ospkI6Cf4WU3djAAhN4pHeOj/wd00Ps1Myr5Cf6UYrI45MXxIFusXQ8Wo/gNhJa+lDUwwnuN87Qy3irHUz46vGKWS2JcvYiiNiiI5B43zwBYc+zi/JXLqik2YKxRYzNgH72fOKCPIrfZFhyshdsVIbdexkZoE= ceph-692607c6-eb12-11ee-95d0-87774f69a715 2024-03-26T01:46:33.868 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-26T01:46:37.389 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:37 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:46:37.390 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:37 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd/host:smithi019", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:46:37.390 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:37 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:46:37.390 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:37 smithi019 ceph-mon[24816]: Deploying daemon alertmanager.smithi019 on smithi019 2024-03-26T01:46:37.390 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:37 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/2264080416' entity='client.admin' 2024-03-26T01:46:37.391 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-26T01:46:37.391 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-26T01:46:40.989 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi069 2024-03-26T01:46:40.989 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-26T01:46:40.989 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/etc/ceph/ceph.conf 2024-03-26T01:46:41.007 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-26T01:46:41.007 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:46:41.065 INFO:tasks.cephadm:Adding host smithi069 to orchestrator... 2024-03-26T01:46:41.065 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph orch host add smithi069 2024-03-26T01:46:41.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:40 smithi019 ceph-mon[24816]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:46:41.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:40 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:46:44.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:43 smithi019 ceph-mon[24816]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi069", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:46:44.407 INFO:teuthology.orchestra.run.smithi019.stdout:Added host 'smithi069' with addr '172.21.15.69' 2024-03-26T01:46:45.267 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-26T01:46:45.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:45 smithi019 ceph-mon[24816]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:46:45.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:45 smithi019 ceph-mon[24816]: Deploying cephadm binary to smithi069 2024-03-26T01:46:45.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:45 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:46:45.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:45 smithi019 ceph-mon[24816]: Added host smithi069 2024-03-26T01:46:47.514 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:47 smithi019 ceph-mon[24816]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:46:48.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:48 smithi019 ceph-mon[24816]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:46:49.428 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-26T01:46:49.428 INFO:teuthology.orchestra.run.smithi019.stdout:[{"addr": "172.21.15.19", "hostname": "smithi019", "labels": [], "status": ""}, {"addr": "172.21.15.69", "hostname": "smithi069", "labels": [], "status": ""}] 2024-03-26T01:46:50.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:49 smithi019 ceph-mon[24816]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T01:46:50.865 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-26T01:46:50.865 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph osd crush tunables default 2024-03-26T01:46:51.378 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:51 smithi019 ceph-mon[24816]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:46:51.378 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:51 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:46:51.378 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:51 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi019", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-26T01:46:51.379 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:51 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi019", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-26T01:46:51.379 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:51 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:46:51.379 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:51 smithi019 ceph-mon[24816]: Deploying daemon crash.smithi019 on smithi019 2024-03-26T01:46:53.483 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:53 smithi019 ceph-mon[24816]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:46:54.261 INFO:teuthology.orchestra.run.smithi019.stderr:adjusted tunables profile to default 2024-03-26T01:46:54.514 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:54 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/1170757039' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T01:46:54.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:54 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:46:55.023 INFO:tasks.cephadm:Adding mon.smithi019 on smithi019 2024-03-26T01:46:55.023 INFO:tasks.cephadm:Adding mon.smithi069 on smithi069 2024-03-26T01:46:55.023 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph orch apply mon '2;smithi019:172.21.15.19=smithi019;smithi069:172.21.15.69=smithi069' 2024-03-26T01:46:55.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:55 smithi019 ceph-mon[24816]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:46:55.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:55 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/1170757039' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T01:46:55.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:55 smithi019 ceph-mon[24816]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T01:46:55.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:55 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:46:55.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:55 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:46:55.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:55 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:46:55.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:55 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T01:46:55.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:55 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:46:56.514 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:56 smithi019 ceph-mon[24816]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T01:46:56.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:56 smithi019 ceph-mon[24816]: Deploying daemon grafana.smithi019 on smithi019 2024-03-26T01:46:56.699 INFO:teuthology.orchestra.run.smithi069.stdout:Scheduled mon update... 2024-03-26T01:46:57.249 DEBUG:teuthology.orchestra.run.smithi069:mon.smithi069> sudo journalctl -f -n 0 -u ceph-692607c6-eb12-11ee-95d0-87774f69a715@mon.smithi069.service 2024-03-26T01:46:57.252 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T01:46:57.252 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T01:46:57.514 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:57 smithi019 ceph-mon[24816]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:46:57.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:57 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:46:58.514 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:58 smithi019 ceph-mon[24816]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi019:172.21.15.19=smithi019;smithi069:172.21.15.69=smithi069", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:46:58.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:58 smithi019 ceph-mon[24816]: Saving service mon spec with placement smithi019:172.21.15.19=smithi019;smithi069:172.21.15.69=smithi069;count:2 2024-03-26T01:46:59.140 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-26T01:46:59.140 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":1,"fsid":"692607c6-eb12-11ee-95d0-87774f69a715","modified":"2024-03-26T01:45:24.115584Z","created":"2024-03-26T01:45:24.115584Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi019","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T01:46:59.142 INFO:teuthology.orchestra.run.smithi069.stderr:dumped monmap epoch 1 2024-03-26T01:46:59.514 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:59 smithi019 ceph-mon[24816]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:46:59.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:59 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:46:59.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:46:59 smithi019 ceph-mon[24816]: from='client.? 172.21.15.69:0/1739734383' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T01:47:00.774 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T01:47:00.775 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T01:47:01.514 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:01 smithi019 ceph-mon[24816]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:47:02.571 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-26T01:47:02.571 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":1,"fsid":"692607c6-eb12-11ee-95d0-87774f69a715","modified":"2024-03-26T01:45:24.115584Z","created":"2024-03-26T01:45:24.115584Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi019","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T01:47:02.573 INFO:teuthology.orchestra.run.smithi069.stderr:dumped monmap epoch 1 2024-03-26T01:47:03.514 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:03 smithi019 ceph-mon[24816]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:47:03.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:03 smithi019 ceph-mon[24816]: from='client.? 172.21.15.69:0/3681751930' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T01:47:04.095 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T01:47:04.095 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T01:47:05.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:05 smithi019 ceph-mon[24816]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:47:05.976 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-26T01:47:05.977 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":1,"fsid":"692607c6-eb12-11ee-95d0-87774f69a715","modified":"2024-03-26T01:45:24.115584Z","created":"2024-03-26T01:45:24.115584Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi019","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T01:47:05.978 INFO:teuthology.orchestra.run.smithi069.stderr:dumped monmap epoch 1 2024-03-26T01:47:06.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:06 smithi019 ceph-mon[24816]: from='client.? 172.21.15.69:0/1329186606' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T01:47:07.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:07 smithi019 ceph-mon[24816]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:47:07.567 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T01:47:07.567 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T01:47:09.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:08 smithi019 ceph-mon[24816]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:47:09.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:08 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:47:09.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:08 smithi019 ceph-mon[24816]: Deploying daemon node-exporter.smithi019 on smithi019 2024-03-26T01:47:09.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:08 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:47:09.486 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-26T01:47:09.486 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":1,"fsid":"692607c6-eb12-11ee-95d0-87774f69a715","modified":"2024-03-26T01:45:24.115584Z","created":"2024-03-26T01:45:24.115584Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi019","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T01:47:09.488 INFO:teuthology.orchestra.run.smithi069.stderr:dumped monmap epoch 1 2024-03-26T01:47:10.214 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:09 smithi019 ceph-mon[24816]: from='client.? 172.21.15.69:0/2270428766' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T01:47:10.929 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:10 smithi019 ceph-mon[24816]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:47:11.055 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T01:47:11.055 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T01:47:12.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:12 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:47:12.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:12 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T01:47:12.807 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-26T01:47:12.808 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":1,"fsid":"692607c6-eb12-11ee-95d0-87774f69a715","modified":"2024-03-26T01:45:24.115584Z","created":"2024-03-26T01:45:24.115584Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi019","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T01:47:12.809 INFO:teuthology.orchestra.run.smithi069.stderr:dumped monmap epoch 1 2024-03-26T01:47:13.378 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:13 smithi019 ceph-mon[24816]: from='mgr.14162 172.21.15.19:0/754543038' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T01:47:13.378 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:13 smithi019 ceph-mon[24816]: mgrmap e14: smithi019.ixuvdm(active, since 48s) 2024-03-26T01:47:13.379 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:13 smithi019 ceph-mon[24816]: from='client.? 172.21.15.69:0/926439314' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T01:47:14.373 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T01:47:14.373 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T01:47:16.247 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-26T01:47:16.247 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":1,"fsid":"692607c6-eb12-11ee-95d0-87774f69a715","modified":"2024-03-26T01:45:24.115584Z","created":"2024-03-26T01:45:24.115584Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi019","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T01:47:16.248 INFO:teuthology.orchestra.run.smithi069.stderr:dumped monmap epoch 1 2024-03-26T01:47:16.373 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:16 smithi019 ceph-mon[24816]: from='client.? 172.21.15.69:0/3058184605' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T01:47:17.794 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T01:47:17.794 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T01:47:19.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:18 smithi019 ceph-mon[24816]: Active manager daemon smithi019.ixuvdm restarted 2024-03-26T01:47:19.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:18 smithi019 ceph-mon[24816]: Activating manager daemon smithi019.ixuvdm 2024-03-26T01:47:19.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:18 smithi019 ceph-mon[24816]: osdmap e5: 0 total, 0 up, 0 in 2024-03-26T01:47:19.661 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-26T01:47:19.661 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":1,"fsid":"692607c6-eb12-11ee-95d0-87774f69a715","modified":"2024-03-26T01:45:24.115584Z","created":"2024-03-26T01:45:24.115584Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi019","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T01:47:19.663 INFO:teuthology.orchestra.run.smithi069.stderr:dumped monmap epoch 1 2024-03-26T01:47:20.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:19 smithi019 ceph-mon[24816]: mgrmap e15: smithi019.ixuvdm(active, starting, since 0.631325s) 2024-03-26T01:47:20.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:19 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi019"}]: dispatch 2024-03-26T01:47:20.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:19 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mgr metadata", "who": "smithi019.ixuvdm", "id": "smithi019.ixuvdm"}]: dispatch 2024-03-26T01:47:20.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:19 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T01:47:20.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:19 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T01:47:20.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:19 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T01:47:20.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:19 smithi019 ceph-mon[24816]: Manager daemon smithi019.ixuvdm is now available 2024-03-26T01:47:20.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:19 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:47:20.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:19 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:47:20.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:19 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:47:20.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:19 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:47:20.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:19 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:47:20.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:19 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:47:20.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:19 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/trash_purge_schedule"}]: dispatch 2024-03-26T01:47:20.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:19 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:47:20.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:19 smithi019 ceph-mon[24816]: from='client.? 172.21.15.69:0/2198656670' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T01:47:21.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:20 smithi019 ceph-mon[24816]: mgrmap e16: smithi019.ixuvdm(active, since 1.63346s) 2024-03-26T01:47:21.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:20 smithi019 ceph-mon[24816]: [26/Mar/2024:01:47:20] ENGINE Bus STARTING 2024-03-26T01:47:21.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:20 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:47:21.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:20 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:47:21.219 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T01:47:21.220 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T01:47:21.985 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:21 smithi019 ceph-mon[24816]: [26/Mar/2024:01:47:20] ENGINE Serving on https://172.21.15.19:7150 2024-03-26T01:47:21.985 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:21 smithi019 ceph-mon[24816]: [26/Mar/2024:01:47:20] ENGINE Bus STARTED 2024-03-26T01:47:21.985 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:21 smithi019 ceph-mon[24816]: mgrmap e17: smithi019.ixuvdm(active, since 2s) 2024-03-26T01:47:21.985 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:21 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:47:24.324 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-26T01:47:24.325 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":1,"fsid":"692607c6-eb12-11ee-95d0-87774f69a715","modified":"2024-03-26T01:45:24.115584Z","created":"2024-03-26T01:45:24.115584Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi019","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T01:47:24.325 INFO:teuthology.orchestra.run.smithi069.stderr:dumped monmap epoch 1 2024-03-26T01:47:24.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:24 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:47:24.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:24 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:47:24.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:24 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd/host:smithi019", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:47:24.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:24 smithi019 ceph-mon[24816]: Updating smithi019:/etc/ceph/ceph.conf 2024-03-26T01:47:24.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:24 smithi019 ceph-mon[24816]: from='client.? 172.21.15.69:0/1284992545' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T01:47:26.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:25 smithi019 ceph-mon[24816]: Updating smithi019:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:47:26.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:25 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:47:26.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:25 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:47:26.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:25 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:47:26.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:25 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd/host:smithi069", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:47:26.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:25 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:47:26.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:25 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi069", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-26T01:47:26.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:25 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi069", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-26T01:47:26.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:25 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:47:26.678 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T01:47:26.678 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T01:47:27.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:26 smithi019 ceph-mon[24816]: Deploying daemon crash.smithi069 on smithi069 2024-03-26T01:47:28.901 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-26T01:47:28.901 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":1,"fsid":"692607c6-eb12-11ee-95d0-87774f69a715","modified":"2024-03-26T01:45:24.115584Z","created":"2024-03-26T01:45:24.115584Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi019","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T01:47:28.903 INFO:teuthology.orchestra.run.smithi069.stderr:dumped monmap epoch 1 2024-03-26T01:47:29.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:28 smithi019 ceph-mon[24816]: from='client.? 172.21.15.69:0/2164639549' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T01:47:29.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:28 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:47:29.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:28 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi069.wpbewu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T01:47:29.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:28 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi069.wpbewu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T01:47:29.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:28 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T01:47:29.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:28 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:47:30.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:29 smithi019 ceph-mon[24816]: Deploying daemon mgr.smithi069.wpbewu on smithi069 2024-03-26T01:47:30.586 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T01:47:30.586 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T01:47:32.549 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-26T01:47:32.549 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":1,"fsid":"692607c6-eb12-11ee-95d0-87774f69a715","modified":"2024-03-26T01:45:24.115584Z","created":"2024-03-26T01:45:24.115584Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi019","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T01:47:32.550 INFO:teuthology.orchestra.run.smithi069.stderr:dumped monmap epoch 1 2024-03-26T01:47:33.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:32 smithi019 ceph-mon[24816]: from='client.? 172.21.15.69:0/194598824' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T01:47:34.376 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T01:47:34.376 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T01:47:34.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:34 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:47:34.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:34 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:47:34.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:34 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:47:34.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:34 smithi019 ceph-mon[24816]: Deploying daemon mon.smithi069 on smithi069 2024-03-26T01:47:34.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:34 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:47:37.476 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-26T01:47:37.476 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":1,"fsid":"692607c6-eb12-11ee-95d0-87774f69a715","modified":"2024-03-26T01:45:24.115584Z","created":"2024-03-26T01:45:24.115584Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi019","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T01:47:37.477 INFO:teuthology.orchestra.run.smithi069.stderr:dumped monmap epoch 1 2024-03-26T01:47:38.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:37 smithi019 ceph-mon[24816]: from='client.? 172.21.15.69:0/3988691356' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T01:47:39.201 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-26T01:47:39.202 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T01:47:43.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:42 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi019"}]: dispatch 2024-03-26T01:47:43.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:42 smithi019 ceph-mon[24816]: mon.smithi019 calling monitor election 2024-03-26T01:47:43.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:42 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi069"}]: dispatch 2024-03-26T01:47:43.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:42 smithi019 ceph-mon[24816]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:47:43.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:42 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi069"}]: dispatch 2024-03-26T01:47:43.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:42 smithi019 ceph-mon[24816]: from='mgr.? 172.21.15.69:0/829198310' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi069.wpbewu/crt"}]: dispatch 2024-03-26T01:47:43.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:42 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi069"}]: dispatch 2024-03-26T01:47:43.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:42 smithi019 ceph-mon[24816]: mon.smithi069 calling monitor election 2024-03-26T01:47:43.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:42 smithi019 ceph-mon[24816]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:47:43.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:42 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi069"}]: dispatch 2024-03-26T01:47:43.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:42 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi069"}]: dispatch 2024-03-26T01:47:43.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:42 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi069"}]: dispatch 2024-03-26T01:47:43.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:42 smithi019 ceph-mon[24816]: mon.smithi019 is new leader, mons smithi019,smithi069 in quorum (ranks 0,1) 2024-03-26T01:47:43.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:42 smithi019 ceph-mon[24816]: monmap e2: 2 mons at {smithi019=[v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0],smithi069=[v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0]} 2024-03-26T01:47:43.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:42 smithi019 ceph-mon[24816]: fsmap 2024-03-26T01:47:43.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:42 smithi019 ceph-mon[24816]: osdmap e5: 0 total, 0 up, 0 in 2024-03-26T01:47:43.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:42 smithi019 ceph-mon[24816]: mgrmap e17: smithi019.ixuvdm(active, since 24s) 2024-03-26T01:47:43.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:42 smithi019 ceph-mon[24816]: Standby manager daemon smithi069.wpbewu started 2024-03-26T01:47:43.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:42 smithi019 ceph-mon[24816]: from='mgr.? 172.21.15.69:0/829198310' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T01:47:43.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:42 smithi019 ceph-mon[24816]: overall HEALTH_OK 2024-03-26T01:47:43.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:42 smithi019 ceph-mon[24816]: from='mgr.? 172.21.15.69:0/829198310' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi069.wpbewu/key"}]: dispatch 2024-03-26T01:47:43.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:42 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:47:43.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:42 smithi019 ceph-mon[24816]: from='mgr.? 172.21.15.69:0/829198310' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T01:47:44.107 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-03-26T01:47:44.108 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":2,"fsid":"692607c6-eb12-11ee-95d0-87774f69a715","modified":"2024-03-26T01:47:37.840690Z","created":"2024-03-26T01:45:24.115584Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi019","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi069","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:3300","nonce":0},{"type":"v1","addr":"172.21.15.69:6789","nonce":0}]},"addr":"172.21.15.69:6789/0","public_addr":"172.21.15.69:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-26T01:47:44.108 INFO:teuthology.orchestra.run.smithi069.stderr:dumped monmap epoch 2 2024-03-26T01:47:44.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:43 smithi019 ceph-mon[24816]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:47:44.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:43 smithi019 ceph-mon[24816]: Deploying daemon node-exporter.smithi069 on smithi069 2024-03-26T01:47:44.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:43 smithi019 ceph-mon[24816]: mgrmap e18: smithi019.ixuvdm(active, since 24s), standbys: smithi069.wpbewu 2024-03-26T01:47:44.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:43 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mgr metadata", "who": "smithi069.wpbewu", "id": "smithi069.wpbewu"}]: dispatch 2024-03-26T01:47:44.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:43 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:47:44.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:43 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi069"}]: dispatch 2024-03-26T01:47:44.666 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-26T01:47:44.666 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph config generate-minimal-conf 2024-03-26T01:47:44.976 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:44 smithi019 ceph-mon[24816]: from='client.? 172.21.15.69:0/781348676' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T01:47:46.185 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:45 smithi019 ceph-mon[24816]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:47:46.576 INFO:teuthology.orchestra.run.smithi019.stdout:# minimal ceph.conf for 692607c6-eb12-11ee-95d0-87774f69a715 2024-03-26T01:47:46.576 INFO:teuthology.orchestra.run.smithi019.stdout:[global] 2024-03-26T01:47:46.576 INFO:teuthology.orchestra.run.smithi019.stdout: fsid = 692607c6-eb12-11ee-95d0-87774f69a715 2024-03-26T01:47:46.576 INFO:teuthology.orchestra.run.smithi019.stdout: mon_host = [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-03-26T01:47:47.146 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:46 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/1611732093' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:47:47.168 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-26T01:47:47.168 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-26T01:47:47.168 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T01:47:47.195 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-26T01:47:47.195 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:47:47.264 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-26T01:47:47.264 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T01:47:47.298 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-26T01:47:47.298 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:47:47.367 INFO:tasks.cephadm:Deploying OSDs... 2024-03-26T01:47:47.367 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-26T01:47:47.367 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T01:47:47.382 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T01:47:47.382 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_1 2024-03-26T01:47:47.437 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T01:47:47.437 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T01:47:47.438 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-26T01:47:47.438 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T01:47:47.438 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T01:47:47.438 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-03-26 01:47:08.487810829 +0000 2024-03-26T01:47:47.438 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-03-26 01:44:01.973951531 +0000 2024-03-26T01:47:47.438 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-03-26 01:44:01.973951531 +0000 2024-03-26T01:47:47.438 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-03-26 01:44:01.973951531 +0000 2024-03-26T01:47:47.438 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T01:47:47.503 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-03-26T01:47:47.504 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-03-26T01:47:47.504 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000259604 s, 2.0 MB/s 2024-03-26T01:47:47.505 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T01:47:47.560 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_2 2024-03-26T01:47:47.615 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T01:47:47.615 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T01:47:47.615 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-26T01:47:47.615 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T01:47:47.615 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T01:47:47.615 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-03-26 01:47:08.487810829 +0000 2024-03-26T01:47:47.615 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-03-26 01:44:02.326947878 +0000 2024-03-26T01:47:47.615 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-03-26 01:44:02.326947878 +0000 2024-03-26T01:47:47.615 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-03-26 01:44:02.326947878 +0000 2024-03-26T01:47:47.616 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T01:47:47.680 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-03-26T01:47:47.680 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-03-26T01:47:47.680 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000208347 s, 2.5 MB/s 2024-03-26T01:47:47.681 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T01:47:47.737 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_3 2024-03-26T01:47:47.791 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T01:47:47.791 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T01:47:47.791 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-26T01:47:47.791 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T01:47:47.792 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T01:47:47.792 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-03-26 01:47:08.487810829 +0000 2024-03-26T01:47:47.792 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-03-26 01:44:02.668944339 +0000 2024-03-26T01:47:47.792 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-03-26 01:44:02.668944339 +0000 2024-03-26T01:47:47.792 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-03-26 01:44:02.668944339 +0000 2024-03-26T01:47:47.792 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T01:47:47.857 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-03-26T01:47:47.857 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-03-26T01:47:47.857 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000203695 s, 2.5 MB/s 2024-03-26T01:47:47.858 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T01:47:47.914 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_4 2024-03-26T01:47:47.968 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T01:47:47.968 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T01:47:47.968 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-26T01:47:47.968 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T01:47:47.968 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T01:47:47.968 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-03-26 01:47:08.488810818 +0000 2024-03-26T01:47:47.968 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-03-26 01:44:03.009940810 +0000 2024-03-26T01:47:47.969 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-03-26 01:44:03.009940810 +0000 2024-03-26T01:47:47.969 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-03-26 01:44:03.009940810 +0000 2024-03-26T01:47:47.969 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T01:47:48.034 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-03-26T01:47:48.034 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-03-26T01:47:48.034 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000197463 s, 2.6 MB/s 2024-03-26T01:47:48.035 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T01:47:48.090 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-03-26T01:47:48.090 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T01:47:48.109 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T01:47:48.109 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_1 2024-03-26T01:47:48.166 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T01:47:48.166 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T01:47:48.166 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-26T01:47:48.166 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T01:47:48.166 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T01:47:48.166 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-03-26 01:47:43.467161411 +0000 2024-03-26T01:47:48.166 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-03-26 01:44:01.404354081 +0000 2024-03-26T01:47:48.166 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-03-26 01:44:01.404354081 +0000 2024-03-26T01:47:48.166 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-03-26 01:44:01.404354081 +0000 2024-03-26T01:47:48.167 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T01:47:48.231 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:47:47 smithi069 ceph-mon[27071]: Deploying daemon prometheus.smithi019 on smithi019 2024-03-26T01:47:48.234 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-03-26T01:47:48.234 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-03-26T01:47:48.234 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000293741 s, 1.7 MB/s 2024-03-26T01:47:48.236 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T01:47:48.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:47 smithi019 ceph-mon[24816]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:47:48.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:47 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:47:48.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:47 smithi019 ceph-mon[24816]: Deploying daemon prometheus.smithi019 on smithi019 2024-03-26T01:47:48.293 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_2 2024-03-26T01:47:48.350 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T01:47:48.351 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T01:47:48.351 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-26T01:47:48.351 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T01:47:48.351 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T01:47:48.351 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-03-26 01:47:43.467161411 +0000 2024-03-26T01:47:48.351 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-03-26 01:44:01.938341864 +0000 2024-03-26T01:47:48.351 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-03-26 01:44:01.938341864 +0000 2024-03-26T01:47:48.351 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-03-26 01:44:01.938341864 +0000 2024-03-26T01:47:48.351 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T01:47:48.420 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-03-26T01:47:48.420 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-03-26T01:47:48.420 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000179538 s, 2.9 MB/s 2024-03-26T01:47:48.421 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T01:47:48.478 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_3 2024-03-26T01:47:48.534 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T01:47:48.534 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T01:47:48.534 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-26T01:47:48.534 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T01:47:48.534 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T01:47:48.534 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-03-26 01:47:43.468161388 +0000 2024-03-26T01:47:48.534 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-03-26 01:44:02.277334102 +0000 2024-03-26T01:47:48.534 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-03-26 01:44:02.277334102 +0000 2024-03-26T01:47:48.534 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-03-26 01:44:02.277334102 +0000 2024-03-26T01:47:48.535 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T01:47:48.601 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-03-26T01:47:48.601 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-03-26T01:47:48.601 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000200869 s, 2.5 MB/s 2024-03-26T01:47:48.602 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T01:47:48.659 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_4 2024-03-26T01:47:48.715 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T01:47:48.716 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T01:47:48.716 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-26T01:47:48.716 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T01:47:48.716 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T01:47:48.716 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-03-26 01:47:43.468161388 +0000 2024-03-26T01:47:48.716 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-03-26 01:44:02.630326023 +0000 2024-03-26T01:47:48.716 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-03-26 01:44:02.630326023 +0000 2024-03-26T01:47:48.716 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-03-26 01:44:02.630326023 +0000 2024-03-26T01:47:48.716 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T01:47:48.782 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-03-26T01:47:48.782 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-03-26T01:47:48.782 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000219086 s, 2.3 MB/s 2024-03-26T01:47:48.783 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T01:47:48.840 INFO:tasks.cephadm:Deploying osd.0 on smithi019 with /dev/vg_nvme/lv_4... 2024-03-26T01:47:48.840 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-26T01:47:49.764 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:49 smithi019 ceph-mon[24816]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:47:49.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:49 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:47:50.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:47:49 smithi069 ceph-mon[27071]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:47:50.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:47:49 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:47:51.525 INFO:teuthology.orchestra.run.smithi019.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-26T01:47:51.526 INFO:teuthology.orchestra.run.smithi019.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-26T01:47:51.526 INFO:teuthology.orchestra.run.smithi019.stderr: stderr: 10+0 records in 2024-03-26T01:47:51.526 INFO:teuthology.orchestra.run.smithi019.stderr:10+0 records out 2024-03-26T01:47:51.526 INFO:teuthology.orchestra.run.smithi019.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114711 s, 914 MB/s 2024-03-26T01:47:51.526 INFO:teuthology.orchestra.run.smithi019.stderr:--> Zapping successful for: 2024-03-26T01:47:52.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:51 smithi019 ceph-mon[24816]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:47:52.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:47:51 smithi069 ceph-mon[27071]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:47:52.414 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi019:vg_nvme/lv_4 2024-03-26T01:47:53.758 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:53 smithi019 ceph-mon[24816]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:47:54.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:47:53 smithi069 ceph-mon[27071]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:47:54.764 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:54 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:47:54.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:54 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:47:54.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:54 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:47:55.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:47:54 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:47:55.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:47:54 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:47:55.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:47:54 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:47:56.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:55 smithi019 ceph-mon[24816]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:47:56.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:55 smithi019 ceph-mon[24816]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:47:56.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:47:55 smithi069 ceph-mon[27071]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:47:56.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:47:55 smithi069 ceph-mon[27071]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:47:58.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:57 smithi019 ceph-mon[24816]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:47:58.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:47:57 smithi069 ceph-mon[27071]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:00.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:47:59 smithi019 ceph-mon[24816]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:00.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:47:59 smithi069 ceph-mon[27071]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:02.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:01 smithi019 ceph-mon[24816]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:02.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:01 smithi069 ceph-mon[27071]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:03.764 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:03 smithi019 ceph-mon[24816]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:04.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:03 smithi069 ceph-mon[27071]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:06.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:05 smithi019 ceph-mon[24816]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:06.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:05 smithi069 ceph-mon[27071]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:08.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:07 smithi019 ceph-mon[24816]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:08.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:07 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:08.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:07 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:48:08.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:07 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:48:08.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:07 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:48:08.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:07 smithi069 ceph-mon[27071]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:08.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:07 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:08.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:07 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:48:08.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:07 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:48:08.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:07 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:48:08.977 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:08 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/3819366868' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef5ce16c-4c20-4779-86ce-b6e84ed2a6d7"}]: dispatch 2024-03-26T01:48:08.978 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:08 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/3819366868' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef5ce16c-4c20-4779-86ce-b6e84ed2a6d7"}]': finished 2024-03-26T01:48:08.978 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:08 smithi069 ceph-mon[27071]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T01:48:08.978 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:08 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:48:08.978 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:08 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:08.978 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:08 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/1030084707' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:48:08.978 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:08 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:09.102 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:08 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/3819366868' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef5ce16c-4c20-4779-86ce-b6e84ed2a6d7"}]: dispatch 2024-03-26T01:48:09.102 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:08 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/3819366868' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef5ce16c-4c20-4779-86ce-b6e84ed2a6d7"}]': finished 2024-03-26T01:48:09.102 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:08 smithi019 ceph-mon[24816]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T01:48:09.103 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:08 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:48:09.103 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:08 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:09.103 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:08 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/1030084707' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:48:09.103 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:08 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:10.237 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:09 smithi019 ceph-mon[24816]: Updating smithi069:/etc/ceph/ceph.conf 2024-03-26T01:48:10.237 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:09 smithi019 ceph-mon[24816]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:10.237 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:09 smithi019 ceph-mon[24816]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:48:10.237 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:09 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:10.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:09 smithi069 ceph-mon[27071]: Updating smithi069:/etc/ceph/ceph.conf 2024-03-26T01:48:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:09 smithi069 ceph-mon[27071]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:09 smithi069 ceph-mon[27071]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:48:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:09 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:12.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:11 smithi019 ceph-mon[24816]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:12.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:11 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:12.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:11 smithi019 ceph-mon[24816]: Updating smithi019:/etc/ceph/ceph.conf 2024-03-26T01:48:12.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:11 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:12.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:11 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:12.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:11 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:48:12.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:11 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T01:48:12.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:11 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:48:12.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:11 smithi069 ceph-mon[27071]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:12.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:11 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:12.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:11 smithi069 ceph-mon[27071]: Updating smithi019:/etc/ceph/ceph.conf 2024-03-26T01:48:12.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:11 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:12.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:11 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:12.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:11 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:48:12.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:11 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T01:48:12.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:11 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:48:13.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:12 smithi019 ceph-mon[24816]: Reconfiguring mon.smithi019 (unknown last config time)... 2024-03-26T01:48:13.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:12 smithi019 ceph-mon[24816]: Reconfiguring daemon mon.smithi019 on smithi019 2024-03-26T01:48:13.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:12 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:13.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:12 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi019.ixuvdm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T01:48:13.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:12 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T01:48:13.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:12 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:48:13.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:12 smithi069 ceph-mon[27071]: Reconfiguring mon.smithi019 (unknown last config time)... 2024-03-26T01:48:13.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:12 smithi069 ceph-mon[27071]: Reconfiguring daemon mon.smithi019 on smithi019 2024-03-26T01:48:13.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:12 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:13.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:12 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi019.ixuvdm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T01:48:13.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:12 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T01:48:13.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:12 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:48:14.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:13 smithi019 ceph-mon[24816]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:14.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:13 smithi019 ceph-mon[24816]: Reconfiguring mgr.smithi019.ixuvdm (unknown last config time)... 2024-03-26T01:48:14.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:13 smithi019 ceph-mon[24816]: Reconfiguring daemon mgr.smithi019.ixuvdm on smithi019 2024-03-26T01:48:14.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:13 smithi069 ceph-mon[27071]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:14.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:13 smithi069 ceph-mon[27071]: Reconfiguring mgr.smithi019.ixuvdm (unknown last config time)... 2024-03-26T01:48:14.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:13 smithi069 ceph-mon[27071]: Reconfiguring daemon mgr.smithi019.ixuvdm on smithi019 2024-03-26T01:48:16.164 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:15 smithi019 ceph-mon[24816]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:16.164 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:15 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:16.164 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:15 smithi019 ceph-mon[24816]: Reconfiguring alertmanager.smithi019 (dependencies changed)... 2024-03-26T01:48:16.164 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:15 smithi019 ceph-mon[24816]: Reconfiguring daemon alertmanager.smithi019 on smithi019 2024-03-26T01:48:16.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:15 smithi069 ceph-mon[27071]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:16.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:15 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:16.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:15 smithi069 ceph-mon[27071]: Reconfiguring alertmanager.smithi019 (dependencies changed)... 2024-03-26T01:48:16.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:15 smithi069 ceph-mon[27071]: Reconfiguring daemon alertmanager.smithi019 on smithi019 2024-03-26T01:48:17.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:16 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T01:48:17.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:16 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:48:17.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:16 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T01:48:17.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:16 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:48:17.954 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:17 smithi019 ceph-mon[24816]: Deploying daemon osd.0 on smithi019 2024-03-26T01:48:17.954 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:17 smithi019 ceph-mon[24816]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:18.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:17 smithi069 ceph-mon[27071]: Deploying daemon osd.0 on smithi019 2024-03-26T01:48:18.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:17 smithi069 ceph-mon[27071]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:19.141 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:18 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:48:19.142 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:18 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/trash_purge_schedule"}]: dispatch 2024-03-26T01:48:19.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:18 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:48:19.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:18 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/trash_purge_schedule"}]: dispatch 2024-03-26T01:48:20.142 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:19 smithi019 ceph-mon[24816]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:20.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:19 smithi069 ceph-mon[27071]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:21.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:21 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:21.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:21 smithi069 ceph-mon[27071]: Reconfiguring crash.smithi019 (monmap changed)... 2024-03-26T01:48:21.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:21 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi019", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-26T01:48:21.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:21 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:48:21.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:21 smithi069 ceph-mon[27071]: Reconfiguring daemon crash.smithi019 on smithi019 2024-03-26T01:48:21.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:21 smithi069 ceph-mon[27071]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:21.963 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:21 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:21.963 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:21 smithi019 ceph-mon[24816]: Reconfiguring crash.smithi019 (monmap changed)... 2024-03-26T01:48:21.963 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:21 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi019", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-26T01:48:21.963 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:21 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:48:21.963 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:21 smithi019 ceph-mon[24816]: Reconfiguring daemon crash.smithi019 on smithi019 2024-03-26T01:48:21.964 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:21 smithi019 ceph-mon[24816]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:24.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:23 smithi019 ceph-mon[24816]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:24.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:23 smithi069 ceph-mon[27071]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:25.975 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:25 smithi019 ceph-mon[24816]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:25.976 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:25 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:26.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:25 smithi069 ceph-mon[27071]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:26.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:25 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:28.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:27 smithi019 ceph-mon[24816]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:28.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:27 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:28.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:27 smithi019 ceph-mon[24816]: Reconfiguring grafana.smithi019 (dependencies changed)... 2024-03-26T01:48:28.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:27 smithi019 ceph-mon[24816]: Reconfiguring daemon grafana.smithi019 on smithi019 2024-03-26T01:48:28.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:27 smithi069 ceph-mon[27071]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:28.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:27 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:28.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:27 smithi069 ceph-mon[27071]: Reconfiguring grafana.smithi019 (dependencies changed)... 2024-03-26T01:48:28.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:27 smithi069 ceph-mon[27071]: Reconfiguring daemon grafana.smithi019 on smithi019 2024-03-26T01:48:29.261 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:28 smithi019 ceph-mon[24816]: from='osd.0 [v2:172.21.15.19:6802/3368385646,v1:172.21.15.19:6803/3368385646]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T01:48:29.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:28 smithi069 ceph-mon[27071]: from='osd.0 [v2:172.21.15.19:6802/3368385646,v1:172.21.15.19:6803/3368385646]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T01:48:30.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:29 smithi019 ceph-mon[24816]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:30.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:29 smithi019 ceph-mon[24816]: from='osd.0 [v2:172.21.15.19:6802/3368385646,v1:172.21.15.19:6803/3368385646]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T01:48:30.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:29 smithi019 ceph-mon[24816]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T01:48:30.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:29 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:48:30.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:29 smithi019 ceph-mon[24816]: from='osd.0 [v2:172.21.15.19:6802/3368385646,v1:172.21.15.19:6803/3368385646]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-03-26T01:48:30.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:29 smithi069 ceph-mon[27071]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:30.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:29 smithi069 ceph-mon[27071]: from='osd.0 [v2:172.21.15.19:6802/3368385646,v1:172.21.15.19:6803/3368385646]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T01:48:30.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:29 smithi069 ceph-mon[27071]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T01:48:30.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:29 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:48:30.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:29 smithi069 ceph-mon[27071]: from='osd.0 [v2:172.21.15.19:6802/3368385646,v1:172.21.15.19:6803/3368385646]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-03-26T01:48:31.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:30 smithi019 ceph-mon[24816]: from='osd.0 [v2:172.21.15.19:6802/3368385646,v1:172.21.15.19:6803/3368385646]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-03-26T01:48:31.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:30 smithi019 ceph-mon[24816]: osdmap e8: 1 total, 0 up, 1 in 2024-03-26T01:48:31.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:30 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:48:31.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:30 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:48:31.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:30 smithi069 ceph-mon[27071]: from='osd.0 [v2:172.21.15.19:6802/3368385646,v1:172.21.15.19:6803/3368385646]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-03-26T01:48:31.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:30 smithi069 ceph-mon[27071]: osdmap e8: 1 total, 0 up, 1 in 2024-03-26T01:48:31.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:30 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:48:31.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:30 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:48:32.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:31 smithi019 ceph-mon[24816]: purged_snaps scrub starts 2024-03-26T01:48:32.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:31 smithi019 ceph-mon[24816]: purged_snaps scrub ok 2024-03-26T01:48:32.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:31 smithi019 ceph-mon[24816]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:32.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:31 smithi019 ceph-mon[24816]: osd.0 [v2:172.21.15.19:6802/3368385646,v1:172.21.15.19:6803/3368385646] boot 2024-03-26T01:48:32.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:31 smithi019 ceph-mon[24816]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T01:48:32.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:31 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:48:32.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:31 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:32.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:31 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi069", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-26T01:48:32.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:31 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:48:32.254 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:31 smithi069 ceph-mon[27071]: purged_snaps scrub starts 2024-03-26T01:48:32.255 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:31 smithi069 ceph-mon[27071]: purged_snaps scrub ok 2024-03-26T01:48:32.255 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:31 smithi069 ceph-mon[27071]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:48:32.255 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:31 smithi069 ceph-mon[27071]: osd.0 [v2:172.21.15.19:6802/3368385646,v1:172.21.15.19:6803/3368385646] boot 2024-03-26T01:48:32.255 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:31 smithi069 ceph-mon[27071]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T01:48:32.255 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:31 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:48:32.255 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:31 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:32.255 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:31 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi069", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-26T01:48:32.255 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:31 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:48:33.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:32 smithi019 ceph-mon[24816]: Reconfiguring crash.smithi069 (monmap changed)... 2024-03-26T01:48:33.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:32 smithi019 ceph-mon[24816]: Reconfiguring daemon crash.smithi069 on smithi069 2024-03-26T01:48:33.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:32 smithi019 ceph-mon[24816]: osdmap e10: 1 total, 1 up, 1 in 2024-03-26T01:48:33.274 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:32 smithi069 ceph-mon[27071]: Reconfiguring crash.smithi069 (monmap changed)... 2024-03-26T01:48:33.274 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:32 smithi069 ceph-mon[27071]: Reconfiguring daemon crash.smithi069 on smithi069 2024-03-26T01:48:33.275 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:32 smithi069 ceph-mon[27071]: osdmap e10: 1 total, 1 up, 1 in 2024-03-26T01:48:34.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:33 smithi019 ceph-mon[24816]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:48:34.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:33 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:34.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:33 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi069.wpbewu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T01:48:34.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:33 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T01:48:34.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:33 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:48:34.292 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:33 smithi069 ceph-mon[27071]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:48:34.292 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:33 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:34.292 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:33 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi069.wpbewu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T01:48:34.292 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:33 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T01:48:34.293 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:33 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:48:34.322 INFO:teuthology.orchestra.run.smithi019.stdout:Created osd(s) 0 on host 'smithi019' 2024-03-26T01:48:34.906 DEBUG:teuthology.orchestra.run.smithi019:osd.0> sudo journalctl -f -n 0 -u ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.0.service 2024-03-26T01:48:34.909 INFO:tasks.cephadm:Deploying osd.1 on smithi019 with /dev/vg_nvme/lv_3... 2024-03-26T01:48:34.909 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-26T01:48:35.175 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:34 smithi069 ceph-mon[27071]: Reconfiguring mgr.smithi069.wpbewu (monmap changed)... 2024-03-26T01:48:35.175 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:34 smithi069 ceph-mon[27071]: Reconfiguring daemon mgr.smithi069.wpbewu on smithi069 2024-03-26T01:48:35.175 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:34 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:35.175 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:34 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:35.175 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:34 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:48:35.175 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:34 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T01:48:35.175 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:34 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:48:35.259 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:34 smithi019 ceph-mon[24816]: Reconfiguring mgr.smithi069.wpbewu (monmap changed)... 2024-03-26T01:48:35.259 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:34 smithi019 ceph-mon[24816]: Reconfiguring daemon mgr.smithi069.wpbewu on smithi069 2024-03-26T01:48:35.259 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:34 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:35.259 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:34 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:35.259 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:34 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:48:35.259 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:34 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T01:48:35.259 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:34 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:48:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:35 smithi069 ceph-mon[27071]: Reconfiguring mon.smithi069 (monmap changed)... 2024-03-26T01:48:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:35 smithi069 ceph-mon[27071]: Reconfiguring daemon mon.smithi069 on smithi069 2024-03-26T01:48:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:35 smithi069 ceph-mon[27071]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:48:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:35 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:35 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T01:48:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:35 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.19:9093"}]: dispatch 2024-03-26T01:48:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:35 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:35 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T01:48:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:35 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.19:3000"}]: dispatch 2024-03-26T01:48:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:35 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:35 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T01:48:36.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:35 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.19:9095"}]: dispatch 2024-03-26T01:48:36.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:35 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:36.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:35 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:48:36.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:35 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:48:36.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:35 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:48:36.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:35 smithi019 ceph-mon[24816]: Reconfiguring mon.smithi069 (monmap changed)... 2024-03-26T01:48:36.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:35 smithi019 ceph-mon[24816]: Reconfiguring daemon mon.smithi069 on smithi069 2024-03-26T01:48:36.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:35 smithi019 ceph-mon[24816]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:48:36.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:35 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:36.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:35 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T01:48:36.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:35 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.19:9093"}]: dispatch 2024-03-26T01:48:36.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:35 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:36.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:35 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T01:48:36.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:35 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.19:3000"}]: dispatch 2024-03-26T01:48:36.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:35 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:36.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:35 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T01:48:36.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:35 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.19:9095"}]: dispatch 2024-03-26T01:48:36.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:35 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:36.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:35 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:48:36.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:35 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:48:36.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:35 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:48:36.908 INFO:teuthology.orchestra.run.smithi019.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-26T01:48:36.908 INFO:teuthology.orchestra.run.smithi019.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-26T01:48:36.908 INFO:teuthology.orchestra.run.smithi019.stderr: stderr: 10+0 records in 2024-03-26T01:48:36.908 INFO:teuthology.orchestra.run.smithi019.stderr:10+0 records out 2024-03-26T01:48:36.908 INFO:teuthology.orchestra.run.smithi019.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0155909 s, 673 MB/s 2024-03-26T01:48:36.908 INFO:teuthology.orchestra.run.smithi019.stderr:--> Zapping successful for: 2024-03-26T01:48:37.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:36 smithi019 ceph-mon[24816]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T01:48:37.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:36 smithi019 ceph-mon[24816]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.19:9093"}]: dispatch 2024-03-26T01:48:37.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:36 smithi019 ceph-mon[24816]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T01:48:37.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:36 smithi019 ceph-mon[24816]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.19:3000"}]: dispatch 2024-03-26T01:48:37.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:36 smithi019 ceph-mon[24816]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T01:48:37.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:36 smithi019 ceph-mon[24816]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.19:9095"}]: dispatch 2024-03-26T01:48:37.330 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:36 smithi069 ceph-mon[27071]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T01:48:37.330 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:36 smithi069 ceph-mon[27071]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.19:9093"}]: dispatch 2024-03-26T01:48:37.330 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:36 smithi069 ceph-mon[27071]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T01:48:37.331 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:36 smithi069 ceph-mon[27071]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.19:3000"}]: dispatch 2024-03-26T01:48:37.331 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:36 smithi069 ceph-mon[27071]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T01:48:37.331 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:36 smithi069 ceph-mon[27071]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.19:9095"}]: dispatch 2024-03-26T01:48:38.016 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi019:vg_nvme/lv_3 2024-03-26T01:48:38.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:37 smithi069 ceph-mon[27071]: pgmap v37: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:48:38.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:37 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:38.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:37 smithi019 ceph-mon[24816]: pgmap v37: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:48:38.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:37 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:40.115 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:39 smithi019 ceph-mon[24816]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:48:40.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:39 smithi069 ceph-mon[27071]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:48:41.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:40 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:48:41.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:40 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:48:41.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:40 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:48:41.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:40 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:48:41.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:40 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:48:41.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:40 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:48:42.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:41 smithi019 ceph-mon[24816]: from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:48:42.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:41 smithi019 ceph-mon[24816]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:48:42.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:41 smithi069 ceph-mon[27071]: from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:48:42.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:41 smithi069 ceph-mon[27071]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:48:44.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:43 smithi019 ceph-mon[24816]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:48:44.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:43 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/518787132' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2a2f72d2-165a-4941-a161-35e3a9bcb44b"}]: dispatch 2024-03-26T01:48:44.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:43 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/518787132' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2a2f72d2-165a-4941-a161-35e3a9bcb44b"}]': finished 2024-03-26T01:48:44.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:43 smithi019 ceph-mon[24816]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T01:48:44.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:43 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:48:44.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:43 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:43 smithi069 ceph-mon[27071]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:48:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:43 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/518787132' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2a2f72d2-165a-4941-a161-35e3a9bcb44b"}]: dispatch 2024-03-26T01:48:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:43 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/518787132' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2a2f72d2-165a-4941-a161-35e3a9bcb44b"}]': finished 2024-03-26T01:48:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:43 smithi069 ceph-mon[27071]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T01:48:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:43 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:48:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:43 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:45.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:44 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/1999949262' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:48:45.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:45 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/1999949262' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:48:46.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:46 smithi019 ceph-mon[24816]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:48:46.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:46 smithi069 ceph-mon[27071]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:48:48.096 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:48 smithi019 ceph-mon[24816]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:48:48.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:48 smithi069 ceph-mon[27071]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:48:50.329 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:50 smithi069 ceph-mon[27071]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:48:50.329 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:50 smithi069 ceph-mon[27071]: Detected new or changed devices on smithi019 2024-03-26T01:48:50.329 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:50 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:50.330 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:50 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:48:50.330 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:50 smithi069 ceph-mon[27071]: Adjusting osd_memory_target on smithi019 to 12869M 2024-03-26T01:48:50.330 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:50 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:50.330 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:50 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:50.330 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:50 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:48:50.330 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:50 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:48:50.330 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:50 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:48:50.388 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:50 smithi019 ceph-mon[24816]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:48:50.388 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:50 smithi019 ceph-mon[24816]: Detected new or changed devices on smithi019 2024-03-26T01:48:50.388 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:50 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:50.389 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:50 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:48:50.389 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:50 smithi019 ceph-mon[24816]: Adjusting osd_memory_target on smithi019 to 12869M 2024-03-26T01:48:50.389 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:50 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:50.389 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:50 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:50.389 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:50 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:48:50.389 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:50 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:48:50.389 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:50 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:48:51.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:51 smithi069 ceph-mon[27071]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:48:51.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:51 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:51.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:51 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T01:48:51.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:51 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:48:51.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:51 smithi069 ceph-mon[27071]: Deploying daemon osd.1 on smithi019 2024-03-26T01:48:52.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:51 smithi019 ceph-mon[24816]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:48:52.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:51 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:52.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:51 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T01:48:52.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:51 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:48:52.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:51 smithi019 ceph-mon[24816]: Deploying daemon osd.1 on smithi019 2024-03-26T01:48:54.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:53 smithi019 ceph-mon[24816]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:48:54.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:53 smithi069 ceph-mon[27071]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:48:56.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:55 smithi019 ceph-mon[24816]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:48:56.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:55 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:56.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:55 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:56.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:55 smithi069 ceph-mon[27071]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:48:56.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:55 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:56.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:55 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:57.995 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:57 smithi019 ceph-mon[24816]: pgmap v48: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:48:58.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:57 smithi069 ceph-mon[27071]: pgmap v48: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:48:59.830 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:59 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:59.830 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:59 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:48:59.830 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:59 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:48:59.830 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:59 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:48:59.830 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:48:59 smithi019 ceph-mon[24816]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:48:59.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:59 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:48:59.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:59 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:48:59.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:59 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:48:59.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:59 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:48:59.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:48:59 smithi069 ceph-mon[27071]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:49:01.458 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:01 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:01.459 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:01 smithi019 ceph-mon[24816]: pgmap v50: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:49:01.544 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:01 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:01.545 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:01 smithi069 ceph-mon[27071]: pgmap v50: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:49:02.194 INFO:teuthology.orchestra.run.smithi019.stdout:Created osd(s) 1 on host 'smithi019' 2024-03-26T01:49:02.488 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:02 smithi019 ceph-mon[24816]: from='osd.1 [v2:172.21.15.19:6810/1744269807,v1:172.21.15.19:6811/1744269807]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T01:49:02.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:02 smithi069 ceph-mon[27071]: from='osd.1 [v2:172.21.15.19:6810/1744269807,v1:172.21.15.19:6811/1744269807]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T01:49:02.766 DEBUG:teuthology.orchestra.run.smithi019:osd.1> sudo journalctl -f -n 0 -u ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.1.service 2024-03-26T01:49:02.769 INFO:tasks.cephadm:Deploying osd.2 on smithi019 with /dev/vg_nvme/lv_2... 2024-03-26T01:49:02.769 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-26T01:49:03.355 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:03 smithi019 ceph-mon[24816]: from='osd.1 [v2:172.21.15.19:6810/1744269807,v1:172.21.15.19:6811/1744269807]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T01:49:03.355 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:03 smithi019 ceph-mon[24816]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T01:49:03.355 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:03 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:49:03.356 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:03 smithi019 ceph-mon[24816]: from='osd.1 [v2:172.21.15.19:6810/1744269807,v1:172.21.15.19:6811/1744269807]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-03-26T01:49:03.356 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:03 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:03.356 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:03 smithi019 ceph-mon[24816]: pgmap v52: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:49:03.356 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 01:49:03 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1[39358]: 2024-03-26T01:49:03.196+0000 7fc1eecb4700 -1 osd.1 0 waiting for initial osdmap 2024-03-26T01:49:03.356 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 01:49:03 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1[39358]: 2024-03-26T01:49:03.199+0000 7fc1e7e48700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T01:49:03.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:03 smithi069 ceph-mon[27071]: from='osd.1 [v2:172.21.15.19:6810/1744269807,v1:172.21.15.19:6811/1744269807]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T01:49:03.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:03 smithi069 ceph-mon[27071]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T01:49:03.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:03 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:49:03.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:03 smithi069 ceph-mon[27071]: from='osd.1 [v2:172.21.15.19:6810/1744269807,v1:172.21.15.19:6811/1744269807]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-03-26T01:49:03.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:03 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:03.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:03 smithi069 ceph-mon[27071]: pgmap v52: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:49:04.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:04 smithi019 ceph-mon[24816]: from='osd.1 [v2:172.21.15.19:6810/1744269807,v1:172.21.15.19:6811/1744269807]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-03-26T01:49:04.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:04 smithi019 ceph-mon[24816]: osdmap e13: 2 total, 1 up, 2 in 2024-03-26T01:49:04.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:04 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:49:04.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:04 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:49:04.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:04 smithi069 ceph-mon[27071]: from='osd.1 [v2:172.21.15.19:6810/1744269807,v1:172.21.15.19:6811/1744269807]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-03-26T01:49:04.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:04 smithi069 ceph-mon[27071]: osdmap e13: 2 total, 1 up, 2 in 2024-03-26T01:49:04.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:04 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:49:04.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:04 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:49:05.355 INFO:teuthology.orchestra.run.smithi019.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-26T01:49:05.356 INFO:teuthology.orchestra.run.smithi019.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-26T01:49:05.356 INFO:teuthology.orchestra.run.smithi019.stderr: stderr: 10+0 records in 2024-03-26T01:49:05.356 INFO:teuthology.orchestra.run.smithi019.stderr:10+0 records out 2024-03-26T01:49:05.356 INFO:teuthology.orchestra.run.smithi019.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0140084 s, 749 MB/s 2024-03-26T01:49:05.356 INFO:teuthology.orchestra.run.smithi019.stderr:--> Zapping successful for: 2024-03-26T01:49:05.457 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:05 smithi019 ceph-mon[24816]: purged_snaps scrub starts 2024-03-26T01:49:05.457 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:05 smithi019 ceph-mon[24816]: purged_snaps scrub ok 2024-03-26T01:49:05.457 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:05 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:49:05.457 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:05 smithi019 ceph-mon[24816]: osd.1 [v2:172.21.15.19:6810/1744269807,v1:172.21.15.19:6811/1744269807] boot 2024-03-26T01:49:05.457 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:05 smithi019 ceph-mon[24816]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T01:49:05.457 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:05 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:49:05.457 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:05 smithi019 ceph-mon[24816]: pgmap v55: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:49:05.509 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:05 smithi069 ceph-mon[27071]: purged_snaps scrub starts 2024-03-26T01:49:05.509 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:05 smithi069 ceph-mon[27071]: purged_snaps scrub ok 2024-03-26T01:49:05.510 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:05 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:49:05.510 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:05 smithi069 ceph-mon[27071]: osd.1 [v2:172.21.15.19:6810/1744269807,v1:172.21.15.19:6811/1744269807] boot 2024-03-26T01:49:05.510 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:05 smithi069 ceph-mon[27071]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T01:49:05.510 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:05 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:49:05.510 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:05 smithi069 ceph-mon[27071]: pgmap v55: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:49:05.940 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi019:vg_nvme/lv_2 2024-03-26T01:49:06.486 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:06 smithi019 ceph-mon[24816]: osdmap e15: 2 total, 2 up, 2 in 2024-03-26T01:49:06.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:06 smithi069 ceph-mon[27071]: osdmap e15: 2 total, 2 up, 2 in 2024-03-26T01:49:07.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:07 smithi019 ceph-mon[24816]: pgmap v57: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:49:07.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:07 smithi069 ceph-mon[27071]: pgmap v57: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:49:08.651 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:08 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:08.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:08 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:09.514 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:09 smithi019 ceph-mon[24816]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:49:09.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:09 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:49:09.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:09 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:49:09.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:09 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:49:09.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:09 smithi019 ceph-mon[24816]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:49:09.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:09 smithi069 ceph-mon[27071]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:49:09.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:09 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:49:09.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:09 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:49:09.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:09 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:49:09.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:09 smithi069 ceph-mon[27071]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:49:11.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:11 smithi069 ceph-mon[27071]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:49:12.010 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:11 smithi019 ceph-mon[24816]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:49:13.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:13 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:13.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:13 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:49:13.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:13 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:49:13.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:13 smithi019 ceph-mon[24816]: Adjusting osd_memory_target on smithi019 to 6434M 2024-03-26T01:49:13.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:13 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:13.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:13 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:13.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:13 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:49:13.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:13 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:49:13.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:13 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:49:13.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:13 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:13.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:13 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:49:13.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:13 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:49:13.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:13 smithi069 ceph-mon[27071]: Adjusting osd_memory_target on smithi019 to 6434M 2024-03-26T01:49:13.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:13 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:13.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:13 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:13.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:13 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:49:13.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:13 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:49:13.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:13 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:49:14.353 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:14 smithi019 ceph-mon[24816]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:49:14.353 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:14 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/3179649580' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b59081e7-53c6-46df-b66e-44f5cf2a79d6"}]: dispatch 2024-03-26T01:49:14.353 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:14 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/3179649580' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b59081e7-53c6-46df-b66e-44f5cf2a79d6"}]': finished 2024-03-26T01:49:14.353 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:14 smithi019 ceph-mon[24816]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T01:49:14.354 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:14 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:49:14.354 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:14 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/2604014737' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:49:14.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:14 smithi069 ceph-mon[27071]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:49:14.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:14 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/3179649580' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b59081e7-53c6-46df-b66e-44f5cf2a79d6"}]: dispatch 2024-03-26T01:49:14.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:14 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/3179649580' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b59081e7-53c6-46df-b66e-44f5cf2a79d6"}]': finished 2024-03-26T01:49:14.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:14 smithi069 ceph-mon[27071]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T01:49:14.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:14 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:49:14.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:14 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/2604014737' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:49:16.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:16 smithi019 ceph-mon[24816]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:49:16.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:16 smithi069 ceph-mon[27071]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:49:18.003 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:17 smithi019 ceph-mon[24816]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:49:18.003 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:17 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:18.003 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:17 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:18.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:17 smithi069 ceph-mon[27071]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:49:18.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:17 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:18.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:17 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:19.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:18 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:49:19.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:18 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/trash_purge_schedule"}]: dispatch 2024-03-26T01:49:19.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:18 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:49:19.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:18 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/trash_purge_schedule"}]: dispatch 2024-03-26T01:49:20.051 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:19 smithi019 ceph-mon[24816]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:49:20.051 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:19 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T01:49:20.051 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:19 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:49:20.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:19 smithi069 ceph-mon[27071]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:49:20.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:19 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T01:49:20.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:19 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:49:21.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:20 smithi019 ceph-mon[24816]: Deploying daemon osd.2 on smithi019 2024-03-26T01:49:21.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:20 smithi069 ceph-mon[27071]: Deploying daemon osd.2 on smithi019 2024-03-26T01:49:22.041 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:21 smithi019 ceph-mon[24816]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:49:22.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:21 smithi069 ceph-mon[27071]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:49:24.007 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:23 smithi019 ceph-mon[24816]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:49:24.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:23 smithi069 ceph-mon[27071]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:49:25.644 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:25 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:26.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:25 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:49:26.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:25 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:49:26.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:25 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:49:26.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:25 smithi019 ceph-mon[24816]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:49:26.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:25 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:26.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:25 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:49:26.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:25 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:49:26.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:25 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:49:26.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:25 smithi069 ceph-mon[27071]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:49:27.525 INFO:teuthology.orchestra.run.smithi019.stdout:Created osd(s) 2 on host 'smithi019' 2024-03-26T01:49:28.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:27 smithi069 ceph-mon[27071]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:49:28.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:27 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:28.182 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:27 smithi019 ceph-mon[24816]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:49:28.182 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:27 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:28.516 DEBUG:teuthology.orchestra.run.smithi019:osd.2> sudo journalctl -f -n 0 -u ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.2.service 2024-03-26T01:49:28.518 INFO:tasks.cephadm:Deploying osd.3 on smithi019 with /dev/vg_nvme/lv_1... 2024-03-26T01:49:28.518 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-26T01:49:29.019 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:28 smithi019 ceph-mon[24816]: from='osd.2 [v2:172.21.15.19:6818/879092529,v1:172.21.15.19:6819/879092529]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T01:49:29.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:28 smithi069 ceph-mon[27071]: from='osd.2 [v2:172.21.15.19:6818/879092529,v1:172.21.15.19:6819/879092529]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T01:49:30.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:29 smithi019 ceph-mon[24816]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:49:30.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:29 smithi019 ceph-mon[24816]: from='osd.2 [v2:172.21.15.19:6818/879092529,v1:172.21.15.19:6819/879092529]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T01:49:30.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:29 smithi019 ceph-mon[24816]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T01:49:30.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:29 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:49:30.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:29 smithi019 ceph-mon[24816]: from='osd.2 [v2:172.21.15.19:6818/879092529,v1:172.21.15.19:6819/879092529]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-03-26T01:49:30.015 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 01:49:29 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2[43080]: 2024-03-26T01:49:29.778+0000 7fa1a4a76700 -1 osd.2 0 waiting for initial osdmap 2024-03-26T01:49:30.015 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 01:49:29 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2[43080]: 2024-03-26T01:49:29.782+0000 7fa19cc08700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T01:49:30.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:29 smithi069 ceph-mon[27071]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:49:30.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:29 smithi069 ceph-mon[27071]: from='osd.2 [v2:172.21.15.19:6818/879092529,v1:172.21.15.19:6819/879092529]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T01:49:30.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:29 smithi069 ceph-mon[27071]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T01:49:30.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:29 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:49:30.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:29 smithi069 ceph-mon[27071]: from='osd.2 [v2:172.21.15.19:6818/879092529,v1:172.21.15.19:6819/879092529]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-03-26T01:49:31.215 INFO:teuthology.orchestra.run.smithi019.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-26T01:49:31.216 INFO:teuthology.orchestra.run.smithi019.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-26T01:49:31.216 INFO:teuthology.orchestra.run.smithi019.stderr: stderr: 10+0 records in 2024-03-26T01:49:31.216 INFO:teuthology.orchestra.run.smithi019.stderr:10+0 records out 2024-03-26T01:49:31.216 INFO:teuthology.orchestra.run.smithi019.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123846 s, 847 MB/s 2024-03-26T01:49:31.216 INFO:teuthology.orchestra.run.smithi019.stderr:--> Zapping successful for: 2024-03-26T01:49:31.216 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:31 smithi019 ceph-mon[24816]: purged_snaps scrub starts 2024-03-26T01:49:31.216 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:31 smithi019 ceph-mon[24816]: purged_snaps scrub ok 2024-03-26T01:49:31.216 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:31 smithi019 ceph-mon[24816]: from='osd.2 [v2:172.21.15.19:6818/879092529,v1:172.21.15.19:6819/879092529]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-03-26T01:49:31.216 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:31 smithi019 ceph-mon[24816]: osdmap e18: 3 total, 2 up, 3 in 2024-03-26T01:49:31.216 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:31 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:49:31.216 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:31 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:49:31.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:31 smithi069 ceph-mon[27071]: purged_snaps scrub starts 2024-03-26T01:49:31.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:31 smithi069 ceph-mon[27071]: purged_snaps scrub ok 2024-03-26T01:49:31.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:31 smithi069 ceph-mon[27071]: from='osd.2 [v2:172.21.15.19:6818/879092529,v1:172.21.15.19:6819/879092529]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-03-26T01:49:31.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:31 smithi069 ceph-mon[27071]: osdmap e18: 3 total, 2 up, 3 in 2024-03-26T01:49:31.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:31 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:49:31.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:31 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:49:32.171 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:32 smithi019 ceph-mon[24816]: pgmap v72: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:49:32.171 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:32 smithi019 ceph-mon[24816]: osd.2 [v2:172.21.15.19:6818/879092529,v1:172.21.15.19:6819/879092529] boot 2024-03-26T01:49:32.171 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:32 smithi019 ceph-mon[24816]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T01:49:32.171 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:32 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:49:32.171 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:32 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-26T01:49:32.171 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:32 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-26T01:49:32.171 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:32 smithi019 ceph-mon[24816]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T01:49:32.171 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:32 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T01:49:32.196 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi019:vg_nvme/lv_1 2024-03-26T01:49:32.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:32 smithi069 ceph-mon[27071]: pgmap v72: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:49:32.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:32 smithi069 ceph-mon[27071]: osd.2 [v2:172.21.15.19:6818/879092529,v1:172.21.15.19:6819/879092529] boot 2024-03-26T01:49:32.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:32 smithi069 ceph-mon[27071]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T01:49:32.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:32 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:49:32.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:32 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-26T01:49:32.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:32 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-26T01:49:32.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:32 smithi069 ceph-mon[27071]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T01:49:32.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:32 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T01:49:33.429 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 01:49:33 smithi019 sudo[44365]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T01:49:33.429 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 01:49:33 smithi019 sudo[44365]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T01:49:33.429 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 01:49:33 smithi019 sudo[44365]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:49:33.709 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 01:49:33 smithi019 sudo[44381]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T01:49:33.710 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 01:49:33 smithi019 sudo[44381]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T01:49:33.710 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 01:49:33 smithi019 sudo[44381]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:49:33.976 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 01:49:33 smithi019 sudo[44387]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T01:49:33.976 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 01:49:33 smithi019 sudo[44387]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T01:49:33.976 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 01:49:33 smithi019 sudo[44387]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:49:33.976 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:33 smithi019 ceph-mon[24816]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:49:33.977 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:33 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T01:49:33.977 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:33 smithi019 ceph-mon[24816]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T01:49:33.977 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:33 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:34.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:33 smithi069 ceph-mon[27071]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:49:34.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:33 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T01:49:34.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:33 smithi069 ceph-mon[27071]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T01:49:34.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:33 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:34.236 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 01:49:33 smithi019 sudo[44390]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T01:49:34.236 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 01:49:33 smithi019 sudo[44390]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T01:49:34.236 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 01:49:33 smithi019 sudo[44390]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:49:34.503 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 01:49:34 smithi019 sudo[44411]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T01:49:34.503 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 01:49:34 smithi019 sudo[44411]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T01:49:34.503 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 01:49:34 smithi019 sudo[44411]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:49:34.764 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 01:49:34 smithi019 sudo[44424]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T01:49:34.765 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 01:49:34 smithi019 sudo[44424]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T01:49:34.765 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 01:49:34 smithi019 sudo[44424]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:49:35.035 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:34 smithi019 sudo[44471]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T01:49:35.035 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:34 smithi019 sudo[44471]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T01:49:35.035 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:34 smithi019 sudo[44471]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:49:35.296 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:35 smithi069 ceph-mon[27071]: osdmap e22: 3 total, 3 up, 3 in 2024-03-26T01:49:35.296 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:35 smithi069 ceph-mon[27071]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T01:49:35.296 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:35 smithi069 sudo[29400]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T01:49:35.296 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:35 smithi069 sudo[29400]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T01:49:35.296 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:35 smithi069 sudo[29400]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:49:35.372 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:35 smithi019 sudo[44501]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T01:49:35.372 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:35 smithi019 sudo[44501]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T01:49:35.372 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:35 smithi019 sudo[44501]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:49:35.372 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:35 smithi019 ceph-mon[24816]: osdmap e22: 3 total, 3 up, 3 in 2024-03-26T01:49:35.372 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:35 smithi019 ceph-mon[24816]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T01:49:35.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:35 smithi069 sudo[29403]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T01:49:35.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:35 smithi069 sudo[29403]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T01:49:35.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:35 smithi069 sudo[29403]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:49:36.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:36 smithi069 ceph-mon[27071]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:49:36.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:36 smithi069 ceph-mon[27071]: from='client.14296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:49:36.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:36 smithi069 ceph-mon[27071]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T01:49:36.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:36 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi019"}]: dispatch 2024-03-26T01:49:36.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:36 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi069"}]: dispatch 2024-03-26T01:49:36.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:36 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:49:36.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:36 smithi069 ceph-mon[27071]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T01:49:36.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:36 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi019"}]: dispatch 2024-03-26T01:49:36.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:36 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi069"}]: dispatch 2024-03-26T01:49:36.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:36 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:49:36.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:36 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:49:36.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:36 smithi069 ceph-mon[27071]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T01:49:36.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:36 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:36.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:36 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:36.438 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:36 smithi019 ceph-mon[24816]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:49:36.461 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:36 smithi019 ceph-mon[24816]: from='client.14296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:49:36.461 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:36 smithi019 ceph-mon[24816]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T01:49:36.461 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:36 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi019"}]: dispatch 2024-03-26T01:49:36.461 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:36 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi069"}]: dispatch 2024-03-26T01:49:36.461 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:36 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:49:36.461 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:36 smithi019 ceph-mon[24816]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T01:49:36.461 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:36 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi019"}]: dispatch 2024-03-26T01:49:36.461 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:36 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi069"}]: dispatch 2024-03-26T01:49:36.461 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:36 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:49:36.462 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:36 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:49:36.462 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:36 smithi019 ceph-mon[24816]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T01:49:36.462 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:36 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:36.462 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:36 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:37.270 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:37 smithi019 ceph-mon[24816]: mgrmap e19: smithi019.ixuvdm(active, since 2m), standbys: smithi069.wpbewu 2024-03-26T01:49:37.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:37 smithi069 ceph-mon[27071]: mgrmap e19: smithi019.ixuvdm(active, since 2m), standbys: smithi069.wpbewu 2024-03-26T01:49:38.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:38 smithi069 ceph-mon[27071]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:49:38.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:38 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/1537547579' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1031e82e-75ee-484d-aa7a-2ee1d697bdc5"}]: dispatch 2024-03-26T01:49:38.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:38 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/1537547579' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1031e82e-75ee-484d-aa7a-2ee1d697bdc5"}]': finished 2024-03-26T01:49:38.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:38 smithi069 ceph-mon[27071]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T01:49:38.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:38 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:49:38.514 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:38 smithi019 ceph-mon[24816]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:49:38.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:38 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/1537547579' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1031e82e-75ee-484d-aa7a-2ee1d697bdc5"}]: dispatch 2024-03-26T01:49:38.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:38 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/1537547579' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1031e82e-75ee-484d-aa7a-2ee1d697bdc5"}]': finished 2024-03-26T01:49:38.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:38 smithi019 ceph-mon[24816]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T01:49:38.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:38 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:49:39.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:39 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/628596695' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:49:39.514 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:39 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/628596695' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:49:40.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:40 smithi019 ceph-mon[24816]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:49:40.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:40 smithi069 ceph-mon[27071]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:49:42.337 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:42 smithi019 ceph-mon[24816]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-26T01:49:42.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:42 smithi069 ceph-mon[27071]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-26T01:49:44.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:44 smithi069 ceph-mon[27071]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-26T01:49:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:44 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:44 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:49:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:44 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:49:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:44 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:49:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:44 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:44 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:44 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:49:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:44 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:49:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:44 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:49:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:44 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:44.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:44 smithi019 ceph-mon[24816]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-26T01:49:44.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:44 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:44.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:44 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:49:44.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:44 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:49:44.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:44 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:49:44.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:44 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:44.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:44 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:44.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:44 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:49:44.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:44 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:49:44.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:44 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:49:44.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:44 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:45.326 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:45 smithi019 ceph-mon[24816]: Detected new or changed devices on smithi019 2024-03-26T01:49:45.326 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:45 smithi019 ceph-mon[24816]: Adjusting osd_memory_target on smithi019 to 4289M 2024-03-26T01:49:45.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:45 smithi069 ceph-mon[27071]: Detected new or changed devices on smithi019 2024-03-26T01:49:45.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:45 smithi069 ceph-mon[27071]: Adjusting osd_memory_target on smithi019 to 4289M 2024-03-26T01:49:46.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:46 smithi019 ceph-mon[24816]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-26T01:49:46.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:46 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T01:49:46.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:46 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:49:46.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:46 smithi069 ceph-mon[27071]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-26T01:49:46.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:46 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T01:49:46.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:46 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:49:47.168 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:47 smithi019 ceph-mon[24816]: Deploying daemon osd.3 on smithi019 2024-03-26T01:49:47.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:47 smithi069 ceph-mon[27071]: Deploying daemon osd.3 on smithi019 2024-03-26T01:49:48.118 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:48 smithi019 ceph-mon[24816]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-26T01:49:48.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:48 smithi069 ceph-mon[27071]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-26T01:49:50.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:50 smithi069 ceph-mon[27071]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 70 KiB/s wr, 5 op/s 2024-03-26T01:49:50.432 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:50 smithi019 ceph-mon[24816]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 70 KiB/s wr, 5 op/s 2024-03-26T01:49:51.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:51 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:51.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:51 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:49:51.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:51 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:49:51.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:51 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:49:51.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:51 smithi069 ceph-mon[27071]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-26T01:49:52.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:51 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:52.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:51 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:49:52.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:51 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:49:52.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:51 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:49:52.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:51 smithi019 ceph-mon[24816]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-26T01:49:53.763 INFO:teuthology.orchestra.run.smithi019.stdout:Created osd(s) 3 on host 'smithi019' 2024-03-26T01:49:54.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:53 smithi019 ceph-mon[24816]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:49:54.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:53 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:54.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:53 smithi069 ceph-mon[27071]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:49:54.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:53 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:54.722 DEBUG:teuthology.orchestra.run.smithi019:osd.3> sudo journalctl -f -n 0 -u ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.3.service 2024-03-26T01:49:54.724 INFO:tasks.cephadm:Deploying osd.4 on smithi069 with /dev/vg_nvme/lv_4... 2024-03-26T01:49:54.724 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-26T01:49:55.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:54 smithi019 ceph-mon[24816]: from='osd.3 [v2:172.21.15.19:6826/3406505853,v1:172.21.15.19:6827/3406505853]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T01:49:55.036 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:54 smithi069 ceph-mon[27071]: from='osd.3 [v2:172.21.15.19:6826/3406505853,v1:172.21.15.19:6827/3406505853]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T01:49:55.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:55 smithi069 ceph-mon[27071]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:49:55.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:55 smithi069 ceph-mon[27071]: from='osd.3 [v2:172.21.15.19:6826/3406505853,v1:172.21.15.19:6827/3406505853]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T01:49:55.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:55 smithi069 ceph-mon[27071]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T01:49:55.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:55 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:49:55.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:55 smithi069 ceph-mon[27071]: from='osd.3 [v2:172.21.15.19:6826/3406505853,v1:172.21.15.19:6827/3406505853]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-03-26T01:49:56.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:55 smithi019 ceph-mon[24816]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:49:56.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:55 smithi019 ceph-mon[24816]: from='osd.3 [v2:172.21.15.19:6826/3406505853,v1:172.21.15.19:6827/3406505853]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T01:49:56.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:55 smithi019 ceph-mon[24816]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T01:49:56.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:55 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:49:56.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:55 smithi019 ceph-mon[24816]: from='osd.3 [v2:172.21.15.19:6826/3406505853,v1:172.21.15.19:6827/3406505853]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-03-26T01:49:56.015 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 01:49:55 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3[46353]: 2024-03-26T01:49:55.787+0000 7fcf70b29700 -1 osd.3 0 waiting for initial osdmap 2024-03-26T01:49:56.015 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 01:49:55 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3[46353]: 2024-03-26T01:49:55.797+0000 7fcf6b4c0700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T01:49:56.556 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-26T01:49:56.557 INFO:teuthology.orchestra.run.smithi069.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-26T01:49:56.557 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10+0 records in 2024-03-26T01:49:56.557 INFO:teuthology.orchestra.run.smithi069.stderr:10+0 records out 2024-03-26T01:49:56.557 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110343 s, 950 MB/s 2024-03-26T01:49:56.557 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping successful for: 2024-03-26T01:49:56.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:56 smithi069 ceph-mon[27071]: from='osd.3 [v2:172.21.15.19:6826/3406505853,v1:172.21.15.19:6827/3406505853]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-03-26T01:49:56.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:56 smithi069 ceph-mon[27071]: osdmap e25: 4 total, 3 up, 4 in 2024-03-26T01:49:56.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:56 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:49:56.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:56 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:49:57.084 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:56 smithi019 ceph-mon[24816]: from='osd.3 [v2:172.21.15.19:6826/3406505853,v1:172.21.15.19:6827/3406505853]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-03-26T01:49:57.084 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:56 smithi019 ceph-mon[24816]: osdmap e25: 4 total, 3 up, 4 in 2024-03-26T01:49:57.084 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:56 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:49:57.085 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:56 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:49:57.143 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi069:vg_nvme/lv_4 2024-03-26T01:49:58.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:57 smithi019 ceph-mon[24816]: purged_snaps scrub starts 2024-03-26T01:49:58.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:57 smithi019 ceph-mon[24816]: purged_snaps scrub ok 2024-03-26T01:49:58.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:57 smithi019 ceph-mon[24816]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:49:58.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:57 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:49:58.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:57 smithi019 ceph-mon[24816]: osd.3 [v2:172.21.15.19:6826/3406505853,v1:172.21.15.19:6827/3406505853] boot 2024-03-26T01:49:58.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:57 smithi019 ceph-mon[24816]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T01:49:58.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:57 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:49:58.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:57 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:58.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:57 smithi069 ceph-mon[27071]: purged_snaps scrub starts 2024-03-26T01:49:58.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:57 smithi069 ceph-mon[27071]: purged_snaps scrub ok 2024-03-26T01:49:58.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:57 smithi069 ceph-mon[27071]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:49:58.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:57 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:49:58.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:57 smithi069 ceph-mon[27071]: osd.3 [v2:172.21.15.19:6826/3406505853,v1:172.21.15.19:6827/3406505853] boot 2024-03-26T01:49:58.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:57 smithi069 ceph-mon[27071]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T01:49:58.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:57 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:49:58.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:57 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:49:59.100 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:58 smithi069 ceph-mon[27071]: osdmap e27: 4 total, 4 up, 4 in 2024-03-26T01:49:59.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:58 smithi019 ceph-mon[24816]: osdmap e27: 4 total, 4 up, 4 in 2024-03-26T01:50:00.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:59 smithi019 ceph-mon[24816]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:50:00.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:59 smithi019 ceph-mon[24816]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:50:00.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:59 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:50:00.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:59 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:50:00.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:49:59 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:50:00.028 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:59 smithi069 ceph-mon[27071]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:50:00.028 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:59 smithi069 ceph-mon[27071]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:50:00.028 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:59 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:50:00.028 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:59 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:50:00.028 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:49:59 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:50:01.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:00 smithi069 ceph-mon[27071]: overall HEALTH_OK 2024-03-26T01:50:01.177 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:00 smithi019 ceph-mon[24816]: overall HEALTH_OK 2024-03-26T01:50:02.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:01 smithi069 ceph-mon[27071]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:50:02.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:01 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:02.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:01 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:50:02.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:01 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:50:02.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:01 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:50:02.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:01 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:50:02.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:01 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:02.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:01 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:02.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:01 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:50:02.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:01 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:50:02.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:01 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:50:02.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:01 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:02.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:01 smithi069 ceph-mon[27071]: from='client.? 172.21.15.69:0/3863741959' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12b11e94-3bef-4235-925d-083e55507ba5"}]: dispatch 2024-03-26T01:50:02.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:01 smithi069 ceph-mon[27071]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12b11e94-3bef-4235-925d-083e55507ba5"}]: dispatch 2024-03-26T01:50:02.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:01 smithi069 ceph-mon[27071]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12b11e94-3bef-4235-925d-083e55507ba5"}]': finished 2024-03-26T01:50:02.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:01 smithi069 ceph-mon[27071]: osdmap e28: 5 total, 4 up, 5 in 2024-03-26T01:50:02.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:01 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T01:50:02.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:01 smithi019 ceph-mon[24816]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:50:02.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:01 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:02.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:01 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:50:02.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:01 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:50:02.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:01 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:50:02.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:01 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:50:02.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:01 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:02.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:01 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:02.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:01 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:50:02.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:01 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:50:02.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:01 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:50:02.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:01 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:02.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:01 smithi019 ceph-mon[24816]: from='client.? 172.21.15.69:0/3863741959' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12b11e94-3bef-4235-925d-083e55507ba5"}]: dispatch 2024-03-26T01:50:02.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:01 smithi019 ceph-mon[24816]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12b11e94-3bef-4235-925d-083e55507ba5"}]: dispatch 2024-03-26T01:50:02.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:01 smithi019 ceph-mon[24816]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12b11e94-3bef-4235-925d-083e55507ba5"}]': finished 2024-03-26T01:50:02.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:01 smithi019 ceph-mon[24816]: osdmap e28: 5 total, 4 up, 5 in 2024-03-26T01:50:02.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:01 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T01:50:03.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:02 smithi069 ceph-mon[27071]: Adjusting osd_memory_target on smithi019 to 3217M 2024-03-26T01:50:03.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:02 smithi069 ceph-mon[27071]: from='client.? 172.21.15.69:0/83070843' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:50:03.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:02 smithi019 ceph-mon[24816]: Adjusting osd_memory_target on smithi019 to 3217M 2024-03-26T01:50:03.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:02 smithi019 ceph-mon[24816]: from='client.? 172.21.15.69:0/83070843' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:50:04.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:03 smithi019 ceph-mon[24816]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:50:04.094 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:03 smithi069 ceph-mon[27071]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:50:06.057 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:05 smithi069 ceph-mon[27071]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:50:06.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:05 smithi019 ceph-mon[24816]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:50:08.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:07 smithi069 ceph-mon[27071]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:50:08.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:07 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T01:50:08.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:07 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:50:08.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:07 smithi019 ceph-mon[24816]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:50:08.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:07 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T01:50:08.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:07 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:50:08.999 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:08 smithi069 ceph-mon[27071]: Deploying daemon osd.4 on smithi069 2024-03-26T01:50:09.252 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:08 smithi019 ceph-mon[24816]: Deploying daemon osd.4 on smithi069 2024-03-26T01:50:10.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:09 smithi069 ceph-mon[27071]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:50:10.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:09 smithi019 ceph-mon[24816]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:50:12.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:11 smithi069 ceph-mon[27071]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:50:12.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:11 smithi019 ceph-mon[24816]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:50:13.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:13 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:13.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:13 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:50:13.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:13 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:50:13.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:13 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:50:13.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:13 smithi019 ceph-mon[24816]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:50:13.777 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:13 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:13.777 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:13 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:50:13.777 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:13 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:50:13.777 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:13 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:50:13.777 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:13 smithi069 ceph-mon[27071]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:50:15.450 INFO:teuthology.orchestra.run.smithi069.stdout:Created osd(s) 4 on host 'smithi069' 2024-03-26T01:50:16.025 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:15 smithi069 ceph-mon[27071]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:50:16.025 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:15 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:16.025 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:15 smithi069 ceph-mon[27071]: from='osd.4 [v2:172.21.15.69:6800/1930906476,v1:172.21.15.69:6801/1930906476]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T01:50:16.026 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:15 smithi069 ceph-mon[27071]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T01:50:16.050 DEBUG:teuthology.orchestra.run.smithi069:osd.4> sudo journalctl -f -n 0 -u ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.4.service 2024-03-26T01:50:16.052 INFO:tasks.cephadm:Deploying osd.5 on smithi069 with /dev/vg_nvme/lv_3... 2024-03-26T01:50:16.052 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-26T01:50:16.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:15 smithi019 ceph-mon[24816]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:50:16.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:15 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:16.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:15 smithi019 ceph-mon[24816]: from='osd.4 [v2:172.21.15.69:6800/1930906476,v1:172.21.15.69:6801/1930906476]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T01:50:16.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:15 smithi019 ceph-mon[24816]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T01:50:17.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:17 smithi019 ceph-mon[24816]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T01:50:17.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:17 smithi019 ceph-mon[24816]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T01:50:17.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:17 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T01:50:17.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:17 smithi019 ceph-mon[24816]: from='osd.4 [v2:172.21.15.69:6800/1930906476,v1:172.21.15.69:6801/1930906476]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-03-26T01:50:17.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:17 smithi019 ceph-mon[24816]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-03-26T01:50:17.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:17 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:17.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:17 smithi019 ceph-mon[24816]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:50:17.810 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:17 smithi069 ceph-mon[27071]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T01:50:17.810 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:17 smithi069 ceph-mon[27071]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T01:50:17.810 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:17 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T01:50:17.810 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:17 smithi069 ceph-mon[27071]: from='osd.4 [v2:172.21.15.69:6800/1930906476,v1:172.21.15.69:6801/1930906476]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-03-26T01:50:17.810 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:17 smithi069 ceph-mon[27071]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-03-26T01:50:17.810 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:17 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:17.810 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:17 smithi069 ceph-mon[27071]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:50:17.811 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 01:50:17 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4[31309]: 2024-03-26T01:50:17.457+0000 7f4a1c4d1700 -1 osd.4 0 waiting for initial osdmap 2024-03-26T01:50:17.811 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 01:50:17 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4[31309]: 2024-03-26T01:50:17.461+0000 7f4a14e64700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T01:50:18.428 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-26T01:50:18.428 INFO:teuthology.orchestra.run.smithi069.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-26T01:50:18.428 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10+0 records in 2024-03-26T01:50:18.429 INFO:teuthology.orchestra.run.smithi069.stderr:10+0 records out 2024-03-26T01:50:18.429 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0148135 s, 708 MB/s 2024-03-26T01:50:18.429 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping successful for: 2024-03-26T01:50:18.737 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:18 smithi019 ceph-mon[24816]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-03-26T01:50:18.737 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:18 smithi019 ceph-mon[24816]: osdmap e30: 5 total, 4 up, 5 in 2024-03-26T01:50:18.737 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:18 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T01:50:18.737 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:18 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T01:50:18.737 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:18 smithi019 ceph-mon[24816]: osd.4 [v2:172.21.15.69:6800/1930906476,v1:172.21.15.69:6801/1930906476] boot 2024-03-26T01:50:18.737 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:18 smithi019 ceph-mon[24816]: osdmap e31: 5 total, 5 up, 5 in 2024-03-26T01:50:18.737 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:18 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T01:50:18.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:18 smithi069 ceph-mon[27071]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-03-26T01:50:18.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:18 smithi069 ceph-mon[27071]: osdmap e30: 5 total, 4 up, 5 in 2024-03-26T01:50:18.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:18 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T01:50:18.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:18 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T01:50:18.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:18 smithi069 ceph-mon[27071]: osd.4 [v2:172.21.15.69:6800/1930906476,v1:172.21.15.69:6801/1930906476] boot 2024-03-26T01:50:18.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:18 smithi069 ceph-mon[27071]: osdmap e31: 5 total, 5 up, 5 in 2024-03-26T01:50:18.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:18 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T01:50:19.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:19 smithi069 ceph-mon[27071]: purged_snaps scrub starts 2024-03-26T01:50:19.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:19 smithi069 ceph-mon[27071]: purged_snaps scrub ok 2024-03-26T01:50:19.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:19 smithi069 ceph-mon[27071]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:50:19.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:19 smithi069 ceph-mon[27071]: osdmap e32: 5 total, 5 up, 5 in 2024-03-26T01:50:19.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:19 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:50:19.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:19 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/trash_purge_schedule"}]: dispatch 2024-03-26T01:50:19.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:19 smithi019 ceph-mon[24816]: purged_snaps scrub starts 2024-03-26T01:50:19.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:19 smithi019 ceph-mon[24816]: purged_snaps scrub ok 2024-03-26T01:50:19.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:19 smithi019 ceph-mon[24816]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:50:19.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:19 smithi019 ceph-mon[24816]: osdmap e32: 5 total, 5 up, 5 in 2024-03-26T01:50:19.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:19 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:50:19.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:19 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/trash_purge_schedule"}]: dispatch 2024-03-26T01:50:20.169 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi069:vg_nvme/lv_3 2024-03-26T01:50:21.004 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:20 smithi069 ceph-mon[27071]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T01:50:21.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:20 smithi019 ceph-mon[24816]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T01:50:21.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:21 smithi069 ceph-mon[27071]: pgmap v112: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-26T01:50:21.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:21 smithi069 ceph-mon[27071]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T01:50:21.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:21 smithi069 ceph-mon[27071]: Detected new or changed devices on smithi069 2024-03-26T01:50:21.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:21 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:21.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:21 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:50:21.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:21 smithi069 ceph-mon[27071]: Adjusting osd_memory_target on smithi069 to 15936M 2024-03-26T01:50:21.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:21 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:21.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:21 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:21.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:21 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:50:21.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:21 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:50:21.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:21 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:50:21.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:21 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:22.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:21 smithi019 ceph-mon[24816]: pgmap v112: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-26T01:50:22.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:21 smithi019 ceph-mon[24816]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T01:50:22.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:21 smithi019 ceph-mon[24816]: Detected new or changed devices on smithi069 2024-03-26T01:50:22.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:21 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:22.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:21 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:50:22.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:21 smithi019 ceph-mon[24816]: Adjusting osd_memory_target on smithi069 to 15936M 2024-03-26T01:50:22.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:21 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:22.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:21 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:22.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:21 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:50:22.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:21 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:50:22.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:21 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:50:22.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:21 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:23.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:22 smithi019 ceph-mon[24816]: osdmap e35: 5 total, 5 up, 5 in 2024-03-26T01:50:23.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:22 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:50:23.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:22 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:50:23.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:22 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:50:23.083 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:22 smithi069 ceph-mon[27071]: osdmap e35: 5 total, 5 up, 5 in 2024-03-26T01:50:23.084 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:22 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:50:23.084 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:22 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:50:23.084 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:22 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:50:23.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:23 smithi069 ceph-mon[27071]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:50:23.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:23 smithi069 ceph-mon[27071]: pgmap v115: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-26T01:50:24.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:23 smithi019 ceph-mon[24816]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:50:24.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:23 smithi019 ceph-mon[24816]: pgmap v115: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-26T01:50:25.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:24 smithi019 ceph-mon[24816]: from='client.? 172.21.15.69:0/2800922934' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "87ffd30b-283e-4da7-8d79-e4ab62c503fc"}]: dispatch 2024-03-26T01:50:25.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:24 smithi019 ceph-mon[24816]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "87ffd30b-283e-4da7-8d79-e4ab62c503fc"}]: dispatch 2024-03-26T01:50:25.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:24 smithi019 ceph-mon[24816]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "87ffd30b-283e-4da7-8d79-e4ab62c503fc"}]': finished 2024-03-26T01:50:25.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:24 smithi019 ceph-mon[24816]: osdmap e36: 6 total, 5 up, 6 in 2024-03-26T01:50:25.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:24 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T01:50:25.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:24 smithi069 ceph-mon[27071]: from='client.? 172.21.15.69:0/2800922934' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "87ffd30b-283e-4da7-8d79-e4ab62c503fc"}]: dispatch 2024-03-26T01:50:25.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:24 smithi069 ceph-mon[27071]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "87ffd30b-283e-4da7-8d79-e4ab62c503fc"}]: dispatch 2024-03-26T01:50:25.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:24 smithi069 ceph-mon[27071]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "87ffd30b-283e-4da7-8d79-e4ab62c503fc"}]': finished 2024-03-26T01:50:25.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:24 smithi069 ceph-mon[27071]: osdmap e36: 6 total, 5 up, 6 in 2024-03-26T01:50:25.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:24 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T01:50:26.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:25 smithi019 ceph-mon[24816]: pgmap v117: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-03-26T01:50:26.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:25 smithi019 ceph-mon[24816]: from='client.? 172.21.15.69:0/2366734280' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:50:26.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:25 smithi069 ceph-mon[27071]: pgmap v117: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-03-26T01:50:26.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:25 smithi069 ceph-mon[27071]: from='client.? 172.21.15.69:0/2366734280' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:50:28.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:27 smithi019 ceph-mon[24816]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:50:28.086 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:27 smithi069 ceph-mon[27071]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:50:30.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:29 smithi019 ceph-mon[24816]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:50:30.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:29 smithi069 ceph-mon[27071]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:50:31.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:31 smithi069 ceph-mon[27071]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:50:31.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:31 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T01:50:31.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:31 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:50:31.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:31 smithi069 ceph-mon[27071]: Deploying daemon osd.5 on smithi069 2024-03-26T01:50:32.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:31 smithi019 ceph-mon[24816]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:50:32.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:31 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T01:50:32.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:31 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:50:32.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:31 smithi019 ceph-mon[24816]: Deploying daemon osd.5 on smithi069 2024-03-26T01:50:33.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:33 smithi069 ceph-mon[27071]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:50:34.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:33 smithi019 ceph-mon[24816]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:50:35.850 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:35 smithi069 ceph-mon[27071]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:50:36.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:35 smithi019 ceph-mon[24816]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:50:37.219 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:36 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:37.220 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:36 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:50:37.220 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:36 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:50:37.220 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:36 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:50:37.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:36 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:37.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:36 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:50:37.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:36 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:50:37.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:36 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:50:38.166 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:37 smithi019 ceph-mon[24816]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:50:38.250 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:37 smithi069 ceph-mon[27071]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:50:38.918 INFO:teuthology.orchestra.run.smithi069.stdout:Created osd(s) 5 on host 'smithi069' 2024-03-26T01:50:39.563 DEBUG:teuthology.orchestra.run.smithi069:osd.5> sudo journalctl -f -n 0 -u ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.5.service 2024-03-26T01:50:39.565 INFO:tasks.cephadm:Deploying osd.6 on smithi069 with /dev/vg_nvme/lv_2... 2024-03-26T01:50:39.565 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-26T01:50:40.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:39 smithi069 ceph-mon[27071]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:50:40.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:39 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:40.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:39 smithi069 ceph-mon[27071]: from='osd.5 [v2:172.21.15.69:6808/3561881892,v1:172.21.15.69:6809/3561881892]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T01:50:40.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:39 smithi069 ceph-mon[27071]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T01:50:40.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:39 smithi019 ceph-mon[24816]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:50:40.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:39 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:40.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:39 smithi019 ceph-mon[24816]: from='osd.5 [v2:172.21.15.69:6808/3561881892,v1:172.21.15.69:6809/3561881892]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T01:50:40.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:39 smithi019 ceph-mon[24816]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T01:50:41.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:40 smithi019 ceph-mon[24816]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T01:50:41.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:40 smithi019 ceph-mon[24816]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T01:50:41.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:40 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T01:50:41.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:40 smithi019 ceph-mon[24816]: from='osd.5 [v2:172.21.15.69:6808/3561881892,v1:172.21.15.69:6809/3561881892]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-03-26T01:50:41.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:40 smithi019 ceph-mon[24816]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-03-26T01:50:41.034 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 01:50:40 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[34284]: 2024-03-26T01:50:40.929+0000 7f134db87700 -1 osd.5 0 waiting for initial osdmap 2024-03-26T01:50:41.034 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 01:50:40 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[34284]: 2024-03-26T01:50:40.938+0000 7f134651a700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T01:50:41.035 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:40 smithi069 ceph-mon[27071]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T01:50:41.035 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:40 smithi069 ceph-mon[27071]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T01:50:41.035 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:40 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T01:50:41.035 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:40 smithi069 ceph-mon[27071]: from='osd.5 [v2:172.21.15.69:6808/3561881892,v1:172.21.15.69:6809/3561881892]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-03-26T01:50:41.035 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:40 smithi069 ceph-mon[27071]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-03-26T01:50:42.096 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-26T01:50:42.096 INFO:teuthology.orchestra.run.smithi069.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-26T01:50:42.096 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10+0 records in 2024-03-26T01:50:42.096 INFO:teuthology.orchestra.run.smithi069.stderr:10+0 records out 2024-03-26T01:50:42.096 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122477 s, 856 MB/s 2024-03-26T01:50:42.096 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping successful for: 2024-03-26T01:50:42.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:41 smithi019 ceph-mon[24816]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:50:42.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:41 smithi019 ceph-mon[24816]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-03-26T01:50:42.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:41 smithi019 ceph-mon[24816]: osdmap e38: 6 total, 5 up, 6 in 2024-03-26T01:50:42.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:41 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T01:50:42.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:41 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T01:50:42.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:41 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:42.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:41 smithi069 ceph-mon[27071]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:50:42.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:41 smithi069 ceph-mon[27071]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-03-26T01:50:42.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:41 smithi069 ceph-mon[27071]: osdmap e38: 6 total, 5 up, 6 in 2024-03-26T01:50:42.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:41 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T01:50:42.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:41 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T01:50:42.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:41 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:43.188 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:42 smithi069 ceph-mon[27071]: purged_snaps scrub starts 2024-03-26T01:50:43.188 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:42 smithi069 ceph-mon[27071]: purged_snaps scrub ok 2024-03-26T01:50:43.188 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:42 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T01:50:43.189 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:42 smithi069 ceph-mon[27071]: osd.5 [v2:172.21.15.69:6808/3561881892,v1:172.21.15.69:6809/3561881892] boot 2024-03-26T01:50:43.189 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:42 smithi069 ceph-mon[27071]: osdmap e39: 6 total, 6 up, 6 in 2024-03-26T01:50:43.189 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:42 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T01:50:43.189 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:42 smithi069 ceph-mon[27071]: osdmap e40: 6 total, 6 up, 6 in 2024-03-26T01:50:43.190 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi069:vg_nvme/lv_2 2024-03-26T01:50:43.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:42 smithi019 ceph-mon[24816]: purged_snaps scrub starts 2024-03-26T01:50:43.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:42 smithi019 ceph-mon[24816]: purged_snaps scrub ok 2024-03-26T01:50:43.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:42 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T01:50:43.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:42 smithi019 ceph-mon[24816]: osd.5 [v2:172.21.15.69:6808/3561881892,v1:172.21.15.69:6809/3561881892] boot 2024-03-26T01:50:43.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:42 smithi019 ceph-mon[24816]: osdmap e39: 6 total, 6 up, 6 in 2024-03-26T01:50:43.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:42 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T01:50:43.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:42 smithi019 ceph-mon[24816]: osdmap e40: 6 total, 6 up, 6 in 2024-03-26T01:50:44.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:43 smithi069 ceph-mon[27071]: pgmap v129: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:50:44.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:43 smithi069 ceph-mon[27071]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-26T01:50:44.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:43 smithi069 ceph-mon[27071]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T01:50:44.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:43 smithi019 ceph-mon[24816]: pgmap v129: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:50:44.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:43 smithi019 ceph-mon[24816]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-26T01:50:44.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:43 smithi019 ceph-mon[24816]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T01:50:46.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:45 smithi019 ceph-mon[24816]: pgmap v132: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-26T01:50:46.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:45 smithi019 ceph-mon[24816]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T01:50:46.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:45 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:46.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:45 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:50:46.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:45 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:50:46.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:45 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:46.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:45 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:46.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:45 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:50:46.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:45 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:50:46.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:45 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:50:46.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:45 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:46.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:45 smithi069 ceph-mon[27071]: pgmap v132: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-26T01:50:46.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:45 smithi069 ceph-mon[27071]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T01:50:46.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:45 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:46.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:45 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:50:46.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:45 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:50:46.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:45 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:46.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:45 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:46.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:45 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:50:46.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:45 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:50:46.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:45 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:50:46.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:45 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:47.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:46 smithi019 ceph-mon[24816]: Detected new or changed devices on smithi069 2024-03-26T01:50:47.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:46 smithi019 ceph-mon[24816]: Adjusting osd_memory_target on smithi069 to 7968M 2024-03-26T01:50:47.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:46 smithi019 ceph-mon[24816]: osdmap e43: 6 total, 6 up, 6 in 2024-03-26T01:50:47.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:46 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:50:47.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:46 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:50:47.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:46 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:50:47.336 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:46 smithi069 ceph-mon[27071]: Detected new or changed devices on smithi069 2024-03-26T01:50:47.336 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:46 smithi069 ceph-mon[27071]: Adjusting osd_memory_target on smithi069 to 7968M 2024-03-26T01:50:47.336 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:46 smithi069 ceph-mon[27071]: osdmap e43: 6 total, 6 up, 6 in 2024-03-26T01:50:47.336 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:46 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:50:47.336 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:46 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:50:47.336 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:46 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:50:48.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:47 smithi019 ceph-mon[24816]: from='client.24187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:50:48.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:47 smithi019 ceph-mon[24816]: pgmap v135: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-26T01:50:48.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:47 smithi069 ceph-mon[27071]: from='client.24187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:50:48.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:47 smithi069 ceph-mon[27071]: pgmap v135: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-26T01:50:49.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:48 smithi019 ceph-mon[24816]: from='client.? 172.21.15.69:0/2687984171' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad320d11-c1b3-4dc8-8e60-6fe1bbf7b377"}]: dispatch 2024-03-26T01:50:49.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:48 smithi019 ceph-mon[24816]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad320d11-c1b3-4dc8-8e60-6fe1bbf7b377"}]: dispatch 2024-03-26T01:50:49.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:48 smithi019 ceph-mon[24816]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad320d11-c1b3-4dc8-8e60-6fe1bbf7b377"}]': finished 2024-03-26T01:50:49.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:48 smithi019 ceph-mon[24816]: osdmap e44: 7 total, 6 up, 7 in 2024-03-26T01:50:49.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:48 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T01:50:49.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:48 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:49.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:48 smithi069 ceph-mon[27071]: from='client.? 172.21.15.69:0/2687984171' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad320d11-c1b3-4dc8-8e60-6fe1bbf7b377"}]: dispatch 2024-03-26T01:50:49.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:48 smithi069 ceph-mon[27071]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad320d11-c1b3-4dc8-8e60-6fe1bbf7b377"}]: dispatch 2024-03-26T01:50:49.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:48 smithi069 ceph-mon[27071]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad320d11-c1b3-4dc8-8e60-6fe1bbf7b377"}]': finished 2024-03-26T01:50:49.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:48 smithi069 ceph-mon[27071]: osdmap e44: 7 total, 6 up, 7 in 2024-03-26T01:50:49.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:48 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T01:50:49.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:48 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:50:50.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:49 smithi019 ceph-mon[24816]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-03-26T01:50:50.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:49 smithi019 ceph-mon[24816]: from='client.? 172.21.15.69:0/3042180406' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:50:50.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:49 smithi019 ceph-mon[24816]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-26T01:50:50.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:49 smithi019 ceph-mon[24816]: Cluster is now healthy 2024-03-26T01:50:50.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:49 smithi069 ceph-mon[27071]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-03-26T01:50:50.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:49 smithi069 ceph-mon[27071]: from='client.? 172.21.15.69:0/3042180406' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:50:50.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:49 smithi069 ceph-mon[27071]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-26T01:50:50.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:49 smithi069 ceph-mon[27071]: Cluster is now healthy 2024-03-26T01:50:52.078 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:51 smithi069 ceph-mon[27071]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-26T01:50:52.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:51 smithi019 ceph-mon[24816]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-26T01:50:54.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:53 smithi069 ceph-mon[27071]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-03-26T01:50:54.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:53 smithi019 ceph-mon[24816]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-03-26T01:50:55.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:54 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T01:50:55.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:54 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:50:55.324 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:54 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T01:50:55.324 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:54 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:50:56.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:55 smithi019 ceph-mon[24816]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-26T01:50:56.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:55 smithi019 ceph-mon[24816]: Deploying daemon osd.6 on smithi069 2024-03-26T01:50:56.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:55 smithi069 ceph-mon[27071]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-26T01:50:56.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:55 smithi069 ceph-mon[27071]: Deploying daemon osd.6 on smithi069 2024-03-26T01:50:58.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:57 smithi019 ceph-mon[24816]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-26T01:50:58.325 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:57 smithi069 ceph-mon[27071]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-26T01:51:00.086 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:50:59 smithi069 ceph-mon[27071]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T01:51:00.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:50:59 smithi019 ceph-mon[24816]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T01:51:01.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:01 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:51:01.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:01 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:51:01.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:01 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:51:01.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:01 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:51:01.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:01 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:51:01.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:01 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:51:01.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:01 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:51:01.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:01 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:51:02.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:02 smithi069 ceph-mon[27071]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T01:51:02.514 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:02 smithi019 ceph-mon[24816]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T01:51:03.706 INFO:teuthology.orchestra.run.smithi069.stdout:Created osd(s) 6 on host 'smithi069' 2024-03-26T01:51:04.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:04 smithi069 ceph-mon[27071]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T01:51:04.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:04 smithi069 ceph-mon[27071]: from='osd.6 [v2:172.21.15.69:6816/1173599153,v1:172.21.15.69:6817/1173599153]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T01:51:04.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:04 smithi069 ceph-mon[27071]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T01:51:04.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:04 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:51:04.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:04 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:51:04.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:04 smithi019 ceph-mon[24816]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T01:51:04.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:04 smithi019 ceph-mon[24816]: from='osd.6 [v2:172.21.15.69:6816/1173599153,v1:172.21.15.69:6817/1173599153]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T01:51:04.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:04 smithi019 ceph-mon[24816]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T01:51:04.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:04 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:51:04.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:04 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:51:04.654 DEBUG:teuthology.orchestra.run.smithi069:osd.6> sudo journalctl -f -n 0 -u ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.6.service 2024-03-26T01:51:04.656 INFO:tasks.cephadm:Deploying osd.7 on smithi069 with /dev/vg_nvme/lv_1... 2024-03-26T01:51:04.656 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-26T01:51:05.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:05 smithi069 ceph-mon[27071]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T01:51:05.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:05 smithi069 ceph-mon[27071]: osdmap e45: 7 total, 6 up, 7 in 2024-03-26T01:51:05.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:05 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T01:51:05.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:05 smithi069 ceph-mon[27071]: from='osd.6 [v2:172.21.15.69:6816/1173599153,v1:172.21.15.69:6817/1173599153]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-03-26T01:51:05.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:05 smithi069 ceph-mon[27071]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-03-26T01:51:05.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:05 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:51:05.359 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 01:51:05 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[37326]: 2024-03-26T01:51:05.073+0000 7f057399b700 -1 osd.6 0 waiting for initial osdmap 2024-03-26T01:51:05.359 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 01:51:05 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[37326]: 2024-03-26T01:51:05.080+0000 7f0570336700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T01:51:05.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:05 smithi019 ceph-mon[24816]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T01:51:05.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:05 smithi019 ceph-mon[24816]: osdmap e45: 7 total, 6 up, 7 in 2024-03-26T01:51:05.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:05 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T01:51:05.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:05 smithi019 ceph-mon[24816]: from='osd.6 [v2:172.21.15.69:6816/1173599153,v1:172.21.15.69:6817/1173599153]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-03-26T01:51:05.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:05 smithi019 ceph-mon[24816]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-03-26T01:51:05.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:05 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:51:06.345 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:06 smithi069 ceph-mon[27071]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T01:51:06.345 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:06 smithi069 ceph-mon[27071]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-03-26T01:51:06.346 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:06 smithi069 ceph-mon[27071]: osdmap e46: 7 total, 6 up, 7 in 2024-03-26T01:51:06.346 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:06 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T01:51:06.346 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:06 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T01:51:06.514 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:06 smithi019 ceph-mon[24816]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T01:51:06.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:06 smithi019 ceph-mon[24816]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-03-26T01:51:06.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:06 smithi019 ceph-mon[24816]: osdmap e46: 7 total, 6 up, 7 in 2024-03-26T01:51:06.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:06 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T01:51:06.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:06 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T01:51:07.352 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:07 smithi069 ceph-mon[27071]: purged_snaps scrub starts 2024-03-26T01:51:07.353 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:07 smithi069 ceph-mon[27071]: purged_snaps scrub ok 2024-03-26T01:51:07.353 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:07 smithi069 ceph-mon[27071]: osd.6 [v2:172.21.15.69:6816/1173599153,v1:172.21.15.69:6817/1173599153] boot 2024-03-26T01:51:07.353 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:07 smithi069 ceph-mon[27071]: osdmap e47: 7 total, 7 up, 7 in 2024-03-26T01:51:07.353 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:07 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T01:51:07.353 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:07 smithi069 ceph-mon[27071]: osdmap e48: 7 total, 7 up, 7 in 2024-03-26T01:51:07.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:07 smithi019 ceph-mon[24816]: purged_snaps scrub starts 2024-03-26T01:51:07.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:07 smithi019 ceph-mon[24816]: purged_snaps scrub ok 2024-03-26T01:51:07.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:07 smithi019 ceph-mon[24816]: osd.6 [v2:172.21.15.69:6816/1173599153,v1:172.21.15.69:6817/1173599153] boot 2024-03-26T01:51:07.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:07 smithi019 ceph-mon[24816]: osdmap e47: 7 total, 7 up, 7 in 2024-03-26T01:51:07.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:07 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T01:51:07.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:07 smithi019 ceph-mon[24816]: osdmap e48: 7 total, 7 up, 7 in 2024-03-26T01:51:08.183 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:08 smithi069 ceph-mon[27071]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-26T01:51:08.184 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:08 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:51:08.184 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:08 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:51:08.184 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:08 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:51:08.184 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:08 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:51:08.184 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:08 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:51:08.184 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:08 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:51:08.184 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:08 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:51:08.184 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:08 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:51:08.184 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:08 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:51:08.184 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:08 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:51:08.184 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:08 smithi069 ceph-mon[27071]: osdmap e49: 7 total, 7 up, 7 in 2024-03-26T01:51:08.416 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:08 smithi019 ceph-mon[24816]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-26T01:51:08.416 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:08 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:51:08.416 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:08 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:51:08.416 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:08 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:51:08.416 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:08 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:51:08.417 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:08 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:51:08.417 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:08 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:51:08.417 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:08 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:51:08.417 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:08 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:51:08.417 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:08 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:51:08.417 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:08 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:51:08.417 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:08 smithi019 ceph-mon[24816]: osdmap e49: 7 total, 7 up, 7 in 2024-03-26T01:51:08.661 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-26T01:51:08.661 INFO:teuthology.orchestra.run.smithi069.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-26T01:51:08.661 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10+0 records in 2024-03-26T01:51:08.661 INFO:teuthology.orchestra.run.smithi069.stderr:10+0 records out 2024-03-26T01:51:08.661 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0186009 s, 564 MB/s 2024-03-26T01:51:08.661 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping successful for: 2024-03-26T01:51:09.321 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:09 smithi069 ceph-mon[27071]: Detected new or changed devices on smithi069 2024-03-26T01:51:09.321 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:09 smithi069 ceph-mon[27071]: Adjusting osd_memory_target on smithi069 to 5312M 2024-03-26T01:51:09.345 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi069:vg_nvme/lv_1 2024-03-26T01:51:09.514 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:09 smithi019 ceph-mon[24816]: Detected new or changed devices on smithi069 2024-03-26T01:51:09.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:09 smithi019 ceph-mon[24816]: Adjusting osd_memory_target on smithi069 to 5312M 2024-03-26T01:51:10.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:10 smithi069 ceph-mon[27071]: pgmap v152: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-26T01:51:10.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:10 smithi069 ceph-mon[27071]: osdmap e50: 7 total, 7 up, 7 in 2024-03-26T01:51:10.514 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:10 smithi019 ceph-mon[24816]: pgmap v152: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-26T01:51:10.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:10 smithi019 ceph-mon[24816]: osdmap e50: 7 total, 7 up, 7 in 2024-03-26T01:51:11.330 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:11 smithi069 ceph-mon[27071]: osdmap e51: 7 total, 7 up, 7 in 2024-03-26T01:51:11.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:11 smithi019 ceph-mon[24816]: osdmap e51: 7 total, 7 up, 7 in 2024-03-26T01:51:12.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:12 smithi069 ceph-mon[27071]: pgmap v155: 1 pgs: 1 remapped+peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T01:51:12.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:12 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:51:12.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:12 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:51:12.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:12 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:51:12.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:12 smithi019 ceph-mon[24816]: pgmap v155: 1 pgs: 1 remapped+peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T01:51:12.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:12 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:51:12.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:12 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:51:12.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:12 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:51:13.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:13 smithi069 ceph-mon[27071]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:51:13.514 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:13 smithi019 ceph-mon[24816]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:51:14.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:14 smithi069 ceph-mon[27071]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-26T01:51:14.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:14 smithi069 ceph-mon[27071]: from='client.? 172.21.15.69:0/3615543771' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1882c969-5f76-4d9f-ba58-73fd558d0d0b"}]: dispatch 2024-03-26T01:51:14.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:14 smithi069 ceph-mon[27071]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1882c969-5f76-4d9f-ba58-73fd558d0d0b"}]: dispatch 2024-03-26T01:51:14.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:14 smithi069 ceph-mon[27071]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1882c969-5f76-4d9f-ba58-73fd558d0d0b"}]': finished 2024-03-26T01:51:14.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:14 smithi069 ceph-mon[27071]: osdmap e52: 8 total, 7 up, 8 in 2024-03-26T01:51:14.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:14 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T01:51:14.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:14 smithi019 ceph-mon[24816]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-26T01:51:14.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:14 smithi019 ceph-mon[24816]: from='client.? 172.21.15.69:0/3615543771' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1882c969-5f76-4d9f-ba58-73fd558d0d0b"}]: dispatch 2024-03-26T01:51:14.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:14 smithi019 ceph-mon[24816]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1882c969-5f76-4d9f-ba58-73fd558d0d0b"}]: dispatch 2024-03-26T01:51:14.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:14 smithi019 ceph-mon[24816]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1882c969-5f76-4d9f-ba58-73fd558d0d0b"}]': finished 2024-03-26T01:51:14.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:14 smithi019 ceph-mon[24816]: osdmap e52: 8 total, 7 up, 8 in 2024-03-26T01:51:14.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:14 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T01:51:15.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:15 smithi069 ceph-mon[27071]: from='client.? 172.21.15.69:0/4111645532' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:51:15.514 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:15 smithi019 ceph-mon[24816]: from='client.? 172.21.15.69:0/4111645532' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:51:16.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:16 smithi069 ceph-mon[27071]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-26T01:51:16.514 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:16 smithi019 ceph-mon[24816]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-26T01:51:18.285 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:18 smithi069 ceph-mon[27071]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-26T01:51:18.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:18 smithi019 ceph-mon[24816]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-26T01:51:19.379 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:19 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/trash_purge_schedule"}]: dispatch 2024-03-26T01:51:19.379 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:19 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:51:19.514 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:19 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/trash_purge_schedule"}]: dispatch 2024-03-26T01:51:19.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:19 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:51:20.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:20 smithi069 ceph-mon[27071]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-26T01:51:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:20 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T01:51:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:20 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:51:20.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:20 smithi019 ceph-mon[24816]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-26T01:51:20.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:20 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T01:51:20.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:20 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:51:21.389 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:21 smithi069 ceph-mon[27071]: Deploying daemon osd.7 on smithi069 2024-03-26T01:51:21.514 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:21 smithi019 ceph-mon[24816]: Deploying daemon osd.7 on smithi069 2024-03-26T01:51:22.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:22 smithi069 ceph-mon[27071]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-26T01:51:22.514 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:22 smithi019 ceph-mon[24816]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-26T01:51:24.211 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:24 smithi069 ceph-mon[27071]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T01:51:24.514 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:24 smithi019 ceph-mon[24816]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T01:51:26.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:26 smithi069 ceph-mon[27071]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T01:51:26.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:26 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:51:26.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:26 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:51:26.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:26 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:51:26.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:26 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:51:26.514 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:26 smithi019 ceph-mon[24816]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T01:51:26.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:26 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:51:26.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:26 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:51:26.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:26 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:51:26.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:26 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:51:28.209 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:28 smithi069 ceph-mon[27071]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T01:51:28.514 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:28 smithi019 ceph-mon[24816]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T01:51:28.918 INFO:teuthology.orchestra.run.smithi069.stdout:Created osd(s) 7 on host 'smithi069' 2024-03-26T01:51:29.296 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:29 smithi069 ceph-mon[27071]: from='osd.7 [v2:172.21.15.69:6824/2299762713,v1:172.21.15.69:6825/2299762713]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T01:51:29.296 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:29 smithi069 ceph-mon[27071]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T01:51:29.296 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:29 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:51:29.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:29 smithi019 ceph-mon[24816]: from='osd.7 [v2:172.21.15.69:6824/2299762713,v1:172.21.15.69:6825/2299762713]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T01:51:29.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:29 smithi019 ceph-mon[24816]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T01:51:29.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:29 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:51:29.592 DEBUG:teuthology.orchestra.run.smithi069:osd.7> sudo journalctl -f -n 0 -u ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.7.service 2024-03-26T01:51:29.595 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-26T01:51:29.595 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-26T01:51:30.168 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:30 smithi019 ceph-mon[24816]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T01:51:30.169 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:30 smithi019 ceph-mon[24816]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T01:51:30.169 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:30 smithi019 ceph-mon[24816]: osdmap e53: 8 total, 7 up, 8 in 2024-03-26T01:51:30.169 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:30 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T01:51:30.169 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:30 smithi019 ceph-mon[24816]: from='osd.7 [v2:172.21.15.69:6824/2299762713,v1:172.21.15.69:6825/2299762713]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-03-26T01:51:30.169 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:30 smithi019 ceph-mon[24816]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-03-26T01:51:30.169 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:30 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:51:30.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:30 smithi069 ceph-mon[27071]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T01:51:30.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:30 smithi069 ceph-mon[27071]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T01:51:30.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:30 smithi069 ceph-mon[27071]: osdmap e53: 8 total, 7 up, 8 in 2024-03-26T01:51:30.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:30 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T01:51:30.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:30 smithi069 ceph-mon[27071]: from='osd.7 [v2:172.21.15.69:6824/2299762713,v1:172.21.15.69:6825/2299762713]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-03-26T01:51:30.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:30 smithi069 ceph-mon[27071]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-03-26T01:51:30.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:30 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:51:30.359 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 01:51:30 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[40323]: 2024-03-26T01:51:30.078+0000 7f494f05d700 -1 osd.7 0 waiting for initial osdmap 2024-03-26T01:51:30.360 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 01:51:30 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[40323]: 2024-03-26T01:51:30.087+0000 7f49479f0700 -1 osd.7 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T01:51:31.351 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:31 smithi069 ceph-mon[27071]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-03-26T01:51:31.351 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:31 smithi069 ceph-mon[27071]: osdmap e54: 8 total, 7 up, 8 in 2024-03-26T01:51:31.351 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:31 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T01:51:31.351 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:31 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T01:51:31.351 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:31 smithi069 ceph-mon[27071]: osd.7 [v2:172.21.15.69:6824/2299762713,v1:172.21.15.69:6825/2299762713] boot 2024-03-26T01:51:31.351 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:31 smithi069 ceph-mon[27071]: osdmap e55: 8 total, 8 up, 8 in 2024-03-26T01:51:31.351 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:31 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T01:51:31.384 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:31 smithi019 ceph-mon[24816]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-03-26T01:51:31.384 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:31 smithi019 ceph-mon[24816]: osdmap e54: 8 total, 7 up, 8 in 2024-03-26T01:51:31.384 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:31 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T01:51:31.384 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:31 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T01:51:31.384 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:31 smithi019 ceph-mon[24816]: osd.7 [v2:172.21.15.69:6824/2299762713,v1:172.21.15.69:6825/2299762713] boot 2024-03-26T01:51:31.384 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:31 smithi019 ceph-mon[24816]: osdmap e55: 8 total, 8 up, 8 in 2024-03-26T01:51:31.384 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:31 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T01:51:31.669 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-26T01:51:32.329 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:32 smithi019 ceph-mon[24816]: purged_snaps scrub starts 2024-03-26T01:51:32.330 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:32 smithi019 ceph-mon[24816]: purged_snaps scrub ok 2024-03-26T01:51:32.330 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:32 smithi019 ceph-mon[24816]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T01:51:32.330 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:32 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/3199896089' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T01:51:32.330 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:32 smithi019 ceph-mon[24816]: osdmap e56: 8 total, 8 up, 8 in 2024-03-26T01:51:32.353 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":55,"num_osds":8,"num_up_osds":8,"osd_up_since":1711417891,"num_in_osds":8,"osd_in_since":1711417873,"num_remapped_pgs":1} 2024-03-26T01:51:32.354 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T01:51:32.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:32 smithi069 ceph-mon[27071]: purged_snaps scrub starts 2024-03-26T01:51:32.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:32 smithi069 ceph-mon[27071]: purged_snaps scrub ok 2024-03-26T01:51:32.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:32 smithi069 ceph-mon[27071]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T01:51:32.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:32 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/3199896089' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T01:51:32.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:32 smithi069 ceph-mon[27071]: osdmap e56: 8 total, 8 up, 8 in 2024-03-26T01:51:32.574 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/mon.smithi019/config 2024-03-26T01:51:33.876 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:33 smithi019 ceph-mon[24816]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:51:33.877 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:33 smithi019 ceph-mon[24816]: Detected new or changed devices on smithi069 2024-03-26T01:51:33.877 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:33 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:51:33.877 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:33 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:51:33.877 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:33 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:51:33.877 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:33 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:51:33.877 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:33 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:51:33.877 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:33 smithi019 ceph-mon[24816]: Adjusting osd_memory_target on smithi069 to 3984M 2024-03-26T01:51:33.877 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:33 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:51:33.877 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:33 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:51:33.877 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:33 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:51:33.877 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:33 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:51:33.877 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:33 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:51:33.877 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:33 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:51:33.877 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:33 smithi019 ceph-mon[24816]: osdmap e57: 8 total, 8 up, 8 in 2024-03-26T01:51:34.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:33 smithi069 ceph-mon[27071]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:51:34.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:33 smithi069 ceph-mon[27071]: Detected new or changed devices on smithi069 2024-03-26T01:51:34.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:33 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:51:34.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:33 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:51:34.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:33 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:51:34.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:33 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:51:34.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:33 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:51:34.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:33 smithi069 ceph-mon[27071]: Adjusting osd_memory_target on smithi069 to 3984M 2024-03-26T01:51:34.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:33 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:51:34.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:33 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:51:34.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:33 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:51:34.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:33 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:51:34.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:33 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:51:34.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:33 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:51:34.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:33 smithi069 ceph-mon[27071]: osdmap e57: 8 total, 8 up, 8 in 2024-03-26T01:51:34.293 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-26T01:51:34.293 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":58,"fsid":"692607c6-eb12-11ee-95d0-87774f69a715","created":"2024-03-26T01:45:26.822265+0000","modified":"2024-03-26T01:51:34.086604+0000","last_up_change":"2024-03-26T01:51:31.071833+0000","last_in_change":"2024-03-26T01:51:13.657944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T01:49:31.028573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"ef5ce16c-4c20-4779-86ce-b6e84ed2a6d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":3368385646},{"type":"v1","addr":"172.21.15.19:6803","nonce":3368385646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":3368385646},{"type":"v1","addr":"172.21.15.19:6805","nonce":3368385646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":3368385646},{"type":"v1","addr":"172.21.15.19:6809","nonce":3368385646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":3368385646},{"type":"v1","addr":"172.21.15.19:6807","nonce":3368385646}]},"public_addr":"172.21.15.19:6803/3368385646","cluster_addr":"172.21.15.19:6805/3368385646","heartbeat_back_addr":"172.21.15.19:6809/3368385646","heartbeat_front_addr":"172.21.15.19:6807/3368385646","state":["exists","up"]},{"osd":1,"uuid":"2a2f72d2-165a-4941-a161-35e3a9bcb44b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":1744269807},{"type":"v1","addr":"172.21.15.19:6811","nonce":1744269807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":1744269807},{"type":"v1","addr":"172.21.15.19:6813","nonce":1744269807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":1744269807},{"type":"v1","addr":"172.21.15.19:6817","nonce":1744269807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":1744269807},{"type":"v1","addr":"172.21.15.19:6815","nonce":1744269807}]},"public_addr":"172.21.15.19:6811/1744269807","cluster_addr":"172.21.15.19:6813/1744269807","heartbeat_back_addr":"172.21.15.19:6817/1744269807","heartbeat_front_addr":"172.21.15.19:6815/1744269807","state":["exists","up"]},{"osd":2,"uuid":"b59081e7-53c6-46df-b66e-44f5cf2a79d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":879092529},{"type":"v1","addr":"172.21.15.19:6819","nonce":879092529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":879092529},{"type":"v1","addr":"172.21.15.19:6821","nonce":879092529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":879092529},{"type":"v1","addr":"172.21.15.19:6825","nonce":879092529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":879092529},{"type":"v1","addr":"172.21.15.19:6823","nonce":879092529}]},"public_addr":"172.21.15.19:6819/879092529","cluster_addr":"172.21.15.19:6821/879092529","heartbeat_back_addr":"172.21.15.19:6825/879092529","heartbeat_front_addr":"172.21.15.19:6823/879092529","state":["exists","up"]},{"osd":3,"uuid":"1031e82e-75ee-484d-aa7a-2ee1d697bdc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":3406505853},{"type":"v1","addr":"172.21.15.19:6827","nonce":3406505853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":3406505853},{"type":"v1","addr":"172.21.15.19:6829","nonce":3406505853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":3406505853},{"type":"v1","addr":"172.21.15.19:6833","nonce":3406505853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":3406505853},{"type":"v1","addr":"172.21.15.19:6831","nonce":3406505853}]},"public_addr":"172.21.15.19:6827/3406505853","cluster_addr":"172.21.15.19:6829/3406505853","heartbeat_back_addr":"172.21.15.19:6833/3406505853","heartbeat_front_addr":"172.21.15.19:6831/3406505853","state":["exists","up"]},{"osd":4,"uuid":"12b11e94-3bef-4235-925d-083e55507ba5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1930906476},{"type":"v1","addr":"172.21.15.69:6801","nonce":1930906476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1930906476},{"type":"v1","addr":"172.21.15.69:6803","nonce":1930906476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1930906476},{"type":"v1","addr":"172.21.15.69:6807","nonce":1930906476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1930906476},{"type":"v1","addr":"172.21.15.69:6805","nonce":1930906476}]},"public_addr":"172.21.15.69:6801/1930906476","cluster_addr":"172.21.15.69:6803/1930906476","heartbeat_back_addr":"172.21.15.69:6807/1930906476","heartbeat_front_addr":"172.21.15.69:6805/1930906476","state":["exists","up"]},{"osd":5,"uuid":"87ffd30b-283e-4da7-8d79-e4ab62c503fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":3561881892},{"type":"v1","addr":"172.21.15.69:6809","nonce":3561881892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3561881892},{"type":"v1","addr":"172.21.15.69:6811","nonce":3561881892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3561881892},{"type":"v1","addr":"172.21.15.69:6815","nonce":3561881892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3561881892},{"type":"v1","addr":"172.21.15.69:6813","nonce":3561881892}]},"public_addr":"172.21.15.69:6809/3561881892","cluster_addr":"172.21.15.69:6811/3561881892","heartbeat_back_addr":"172.21.15.69:6815/3561881892","heartbeat_front_addr":"172.21.15.69:6813/3561881892","state":["exists","up"]},{"osd":6,"uuid":"ad320d11-c1b3-4dc8-8e60-6fe1bbf7b377","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1173599153},{"type":"v1","addr":"172.21.15.69:6817","nonce":1173599153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1173599153},{"type":"v1","addr":"172.21.15.69:6819","nonce":1173599153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1173599153},{"type":"v1","addr":"172.21.15.69:6823","nonce":1173599153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1173599153},{"type":"v1","addr":"172.21.15.69:6821","nonce":1173599153}]},"public_addr":"172.21.15.69:6817/1173599153","cluster_addr":"172.21.15.69:6819/1173599153","heartbeat_back_addr":"172.21.15.69:6823/1173599153","heartbeat_front_addr":"172.21.15.69:6821/1173599153","state":["exists","up"]},{"osd":7,"uuid":"1882c969-5f76-4d9f-ba58-73fd558d0d0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":2299762713},{"type":"v1","addr":"172.21.15.69:6825","nonce":2299762713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":2299762713},{"type":"v1","addr":"172.21.15.69:6827","nonce":2299762713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2299762713},{"type":"v1","addr":"172.21.15.69:6831","nonce":2299762713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":2299762713},{"type":"v1","addr":"172.21.15.69:6829","nonce":2299762713}]},"public_addr":"172.21.15.69:6825/2299762713","cluster_addr":"172.21.15.69:6827/2299762713","heartbeat_back_addr":"172.21.15.69:6831/2299762713","heartbeat_front_addr":"172.21.15.69:6829/2299762713","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:48:29.818218+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:49:02.640823+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:49:28.738825+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:49:54.849692+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:50:16.506248+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:50:39.958875+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:51:04.201784+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:51:29.186801+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/3333157499":"2024-03-27T01:47:18.060767+0000","172.21.15.19:0/3485031281":"2024-03-27T01:47:18.060767+0000","172.21.15.19:0/958839096":"2024-03-27T01:47:18.060767+0000","172.21.15.19:0/2658771683":"2024-03-27T01:46:23.501942+0000","172.21.15.19:0/622698413":"2024-03-27T01:46:23.501942+0000","172.21.15.19:6800/2285111467":"2024-03-27T01:47:18.060767+0000","172.21.15.19:6800/771609471":"2024-03-27T01:46:23.501942+0000","172.21.15.19:0/3007239008":"2024-03-27T01:46:23.501942+0000","172.21.15.19:0/1610114816":"2024-03-27T01:45:54.671764+0000","172.21.15.19:6801/771609471":"2024-03-27T01:46:23.501942+0000","172.21.15.19:6800/101591818":"2024-03-27T01:45:54.671764+0000","172.21.15.19:6801/101591818":"2024-03-27T01:45:54.671764+0000","172.21.15.19:6801/2285111467":"2024-03-27T01:47:18.060767+0000","172.21.15.19:0/3047936116":"2024-03-27T01:45:54.671764+0000","172.21.15.19:0/1016723143":"2024-03-27T01:45:54.671764+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T01:51:35.038 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T01:49:31.028573+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-03-26T01:51:35.038 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-26T01:51:35.260 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/mon.smithi019/config 2024-03-26T01:51:35.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:35 smithi019 ceph-mon[24816]: osdmap e58: 8 total, 8 up, 8 in 2024-03-26T01:51:35.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:35 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/3755919856' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T01:51:35.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:35 smithi069 ceph-mon[27071]: osdmap e58: 8 total, 8 up, 8 in 2024-03-26T01:51:35.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:35 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/3755919856' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T01:51:36.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:36 smithi069 ceph-mon[27071]: pgmap v174: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:51:36.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:36 smithi069 ceph-mon[27071]: osdmap e59: 8 total, 8 up, 8 in 2024-03-26T01:51:36.384 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:36 smithi019 ceph-mon[24816]: pgmap v174: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:51:36.384 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:36 smithi019 ceph-mon[24816]: osdmap e59: 8 total, 8 up, 8 in 2024-03-26T01:51:37.111 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 1 2024-03-26T01:51:37.817 INFO:tasks.cephadm:Setting up client nodes... 2024-03-26T01:51:37.818 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T01:51:38.347 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:38 smithi019 ceph-mon[24816]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-03-26T01:51:38.347 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:38 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/2981195130' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T01:51:38.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:38 smithi069 ceph-mon[27071]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-03-26T01:51:38.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:38 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/2981195130' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T01:51:39.937 INFO:teuthology.orchestra.run.smithi019.stdout:[client.0] 2024-03-26T01:51:39.938 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQArKgJmLAOyNxAAmUw9ZO7XMhBkC4yTLbDsWg== 2024-03-26T01:51:40.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:40 smithi069 ceph-mon[27071]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-26T01:51:40.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:40 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/1427057625' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T01:51:40.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:40 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/1427057625' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T01:51:40.514 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:40 smithi019 ceph-mon[24816]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-26T01:51:40.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:40 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/1427057625' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T01:51:40.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:40 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/1427057625' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T01:51:40.756 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-26T01:51:40.756 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-26T01:51:40.756 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-26T01:51:40.799 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-26T01:51:40.799 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-26T01:51:40.799 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph mgr dump --format=json 2024-03-26T01:51:41.059 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/mon.smithi019/config 2024-03-26T01:51:42.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:42 smithi069 ceph-mon[27071]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-26T01:51:42.420 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:42 smithi019 ceph-mon[24816]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-26T01:51:43.007 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-26T01:51:43.580 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":19,"active_gid":14204,"active_name":"smithi019.ixuvdm","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":3693098909},{"type":"v1","addr":"172.21.15.19:6801","nonce":3693098909}]},"active_addr":"172.21.15.19:6801/3693098909","active_change":"2024-03-26T01:47:18.061106+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":14224,"name":"smithi069.wpbewu","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.19:8443/","prometheus":"http://172.21.15.19:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":5,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.19:0","nonce":4212584823}]},{"addrvec":[{"type":"v2","addr":"172.21.15.19:0","nonce":3695554320}]},{"addrvec":[{"type":"v2","addr":"172.21.15.19:0","nonce":3623610734}]},{"addrvec":[{"type":"v2","addr":"172.21.15.19:0","nonce":2436751289}]}]}} 2024-03-26T01:51:43.586 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-26T01:51:43.586 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-26T01:51:43.587 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T01:51:43.809 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/mon.smithi019/config 2024-03-26T01:51:43.930 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:43 smithi019 ceph-mon[24816]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-26T01:51:43.931 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:43 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/3328105450' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T01:51:44.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:43 smithi069 ceph-mon[27071]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-26T01:51:44.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:43 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/3328105450' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T01:51:45.720 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-26T01:51:45.720 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":59,"fsid":"692607c6-eb12-11ee-95d0-87774f69a715","created":"2024-03-26T01:45:26.822265+0000","modified":"2024-03-26T01:51:35.088295+0000","last_up_change":"2024-03-26T01:51:31.071833+0000","last_in_change":"2024-03-26T01:51:13.657944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T01:49:31.028573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"ef5ce16c-4c20-4779-86ce-b6e84ed2a6d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":3368385646},{"type":"v1","addr":"172.21.15.19:6803","nonce":3368385646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":3368385646},{"type":"v1","addr":"172.21.15.19:6805","nonce":3368385646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":3368385646},{"type":"v1","addr":"172.21.15.19:6809","nonce":3368385646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":3368385646},{"type":"v1","addr":"172.21.15.19:6807","nonce":3368385646}]},"public_addr":"172.21.15.19:6803/3368385646","cluster_addr":"172.21.15.19:6805/3368385646","heartbeat_back_addr":"172.21.15.19:6809/3368385646","heartbeat_front_addr":"172.21.15.19:6807/3368385646","state":["exists","up"]},{"osd":1,"uuid":"2a2f72d2-165a-4941-a161-35e3a9bcb44b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":1744269807},{"type":"v1","addr":"172.21.15.19:6811","nonce":1744269807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":1744269807},{"type":"v1","addr":"172.21.15.19:6813","nonce":1744269807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":1744269807},{"type":"v1","addr":"172.21.15.19:6817","nonce":1744269807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":1744269807},{"type":"v1","addr":"172.21.15.19:6815","nonce":1744269807}]},"public_addr":"172.21.15.19:6811/1744269807","cluster_addr":"172.21.15.19:6813/1744269807","heartbeat_back_addr":"172.21.15.19:6817/1744269807","heartbeat_front_addr":"172.21.15.19:6815/1744269807","state":["exists","up"]},{"osd":2,"uuid":"b59081e7-53c6-46df-b66e-44f5cf2a79d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":879092529},{"type":"v1","addr":"172.21.15.19:6819","nonce":879092529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":879092529},{"type":"v1","addr":"172.21.15.19:6821","nonce":879092529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":879092529},{"type":"v1","addr":"172.21.15.19:6825","nonce":879092529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":879092529},{"type":"v1","addr":"172.21.15.19:6823","nonce":879092529}]},"public_addr":"172.21.15.19:6819/879092529","cluster_addr":"172.21.15.19:6821/879092529","heartbeat_back_addr":"172.21.15.19:6825/879092529","heartbeat_front_addr":"172.21.15.19:6823/879092529","state":["exists","up"]},{"osd":3,"uuid":"1031e82e-75ee-484d-aa7a-2ee1d697bdc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":3406505853},{"type":"v1","addr":"172.21.15.19:6827","nonce":3406505853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":3406505853},{"type":"v1","addr":"172.21.15.19:6829","nonce":3406505853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":3406505853},{"type":"v1","addr":"172.21.15.19:6833","nonce":3406505853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":3406505853},{"type":"v1","addr":"172.21.15.19:6831","nonce":3406505853}]},"public_addr":"172.21.15.19:6827/3406505853","cluster_addr":"172.21.15.19:6829/3406505853","heartbeat_back_addr":"172.21.15.19:6833/3406505853","heartbeat_front_addr":"172.21.15.19:6831/3406505853","state":["exists","up"]},{"osd":4,"uuid":"12b11e94-3bef-4235-925d-083e55507ba5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1930906476},{"type":"v1","addr":"172.21.15.69:6801","nonce":1930906476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1930906476},{"type":"v1","addr":"172.21.15.69:6803","nonce":1930906476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1930906476},{"type":"v1","addr":"172.21.15.69:6807","nonce":1930906476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1930906476},{"type":"v1","addr":"172.21.15.69:6805","nonce":1930906476}]},"public_addr":"172.21.15.69:6801/1930906476","cluster_addr":"172.21.15.69:6803/1930906476","heartbeat_back_addr":"172.21.15.69:6807/1930906476","heartbeat_front_addr":"172.21.15.69:6805/1930906476","state":["exists","up"]},{"osd":5,"uuid":"87ffd30b-283e-4da7-8d79-e4ab62c503fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":3561881892},{"type":"v1","addr":"172.21.15.69:6809","nonce":3561881892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3561881892},{"type":"v1","addr":"172.21.15.69:6811","nonce":3561881892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3561881892},{"type":"v1","addr":"172.21.15.69:6815","nonce":3561881892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3561881892},{"type":"v1","addr":"172.21.15.69:6813","nonce":3561881892}]},"public_addr":"172.21.15.69:6809/3561881892","cluster_addr":"172.21.15.69:6811/3561881892","heartbeat_back_addr":"172.21.15.69:6815/3561881892","heartbeat_front_addr":"172.21.15.69:6813/3561881892","state":["exists","up"]},{"osd":6,"uuid":"ad320d11-c1b3-4dc8-8e60-6fe1bbf7b377","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1173599153},{"type":"v1","addr":"172.21.15.69:6817","nonce":1173599153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1173599153},{"type":"v1","addr":"172.21.15.69:6819","nonce":1173599153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1173599153},{"type":"v1","addr":"172.21.15.69:6823","nonce":1173599153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1173599153},{"type":"v1","addr":"172.21.15.69:6821","nonce":1173599153}]},"public_addr":"172.21.15.69:6817/1173599153","cluster_addr":"172.21.15.69:6819/1173599153","heartbeat_back_addr":"172.21.15.69:6823/1173599153","heartbeat_front_addr":"172.21.15.69:6821/1173599153","state":["exists","up"]},{"osd":7,"uuid":"1882c969-5f76-4d9f-ba58-73fd558d0d0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":2299762713},{"type":"v1","addr":"172.21.15.69:6825","nonce":2299762713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":2299762713},{"type":"v1","addr":"172.21.15.69:6827","nonce":2299762713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2299762713},{"type":"v1","addr":"172.21.15.69:6831","nonce":2299762713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":2299762713},{"type":"v1","addr":"172.21.15.69:6829","nonce":2299762713}]},"public_addr":"172.21.15.69:6825/2299762713","cluster_addr":"172.21.15.69:6827/2299762713","heartbeat_back_addr":"172.21.15.69:6831/2299762713","heartbeat_front_addr":"172.21.15.69:6829/2299762713","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:48:29.818218+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:49:02.640823+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:49:28.738825+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:49:54.849692+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:50:16.506248+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:50:39.958875+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:51:04.201784+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:51:29.186801+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/3333157499":"2024-03-27T01:47:18.060767+0000","172.21.15.19:0/3485031281":"2024-03-27T01:47:18.060767+0000","172.21.15.19:0/958839096":"2024-03-27T01:47:18.060767+0000","172.21.15.19:0/2658771683":"2024-03-27T01:46:23.501942+0000","172.21.15.19:0/622698413":"2024-03-27T01:46:23.501942+0000","172.21.15.19:6800/2285111467":"2024-03-27T01:47:18.060767+0000","172.21.15.19:6800/771609471":"2024-03-27T01:46:23.501942+0000","172.21.15.19:0/3007239008":"2024-03-27T01:46:23.501942+0000","172.21.15.19:0/1610114816":"2024-03-27T01:45:54.671764+0000","172.21.15.19:6801/771609471":"2024-03-27T01:46:23.501942+0000","172.21.15.19:6800/101591818":"2024-03-27T01:45:54.671764+0000","172.21.15.19:6801/101591818":"2024-03-27T01:45:54.671764+0000","172.21.15.19:6801/2285111467":"2024-03-27T01:47:18.060767+0000","172.21.15.19:0/3047936116":"2024-03-27T01:45:54.671764+0000","172.21.15.19:0/1016723143":"2024-03-27T01:45:54.671764+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T01:51:46.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:45 smithi019 ceph-mon[24816]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-26T01:51:46.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:45 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/1600301990' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T01:51:46.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:45 smithi069 ceph-mon[27071]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-26T01:51:46.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:45 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/1600301990' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T01:51:46.403 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-26T01:51:46.404 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T01:51:46.627 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/mon.smithi019/config 2024-03-26T01:51:48.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:47 smithi019 ceph-mon[24816]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s, 0 objects/s recovering 2024-03-26T01:51:48.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:47 smithi069 ceph-mon[27071]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s, 0 objects/s recovering 2024-03-26T01:51:48.601 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-26T01:51:48.601 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":59,"fsid":"692607c6-eb12-11ee-95d0-87774f69a715","created":"2024-03-26T01:45:26.822265+0000","modified":"2024-03-26T01:51:35.088295+0000","last_up_change":"2024-03-26T01:51:31.071833+0000","last_in_change":"2024-03-26T01:51:13.657944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T01:49:31.028573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"ef5ce16c-4c20-4779-86ce-b6e84ed2a6d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":3368385646},{"type":"v1","addr":"172.21.15.19:6803","nonce":3368385646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":3368385646},{"type":"v1","addr":"172.21.15.19:6805","nonce":3368385646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":3368385646},{"type":"v1","addr":"172.21.15.19:6809","nonce":3368385646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":3368385646},{"type":"v1","addr":"172.21.15.19:6807","nonce":3368385646}]},"public_addr":"172.21.15.19:6803/3368385646","cluster_addr":"172.21.15.19:6805/3368385646","heartbeat_back_addr":"172.21.15.19:6809/3368385646","heartbeat_front_addr":"172.21.15.19:6807/3368385646","state":["exists","up"]},{"osd":1,"uuid":"2a2f72d2-165a-4941-a161-35e3a9bcb44b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":1744269807},{"type":"v1","addr":"172.21.15.19:6811","nonce":1744269807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":1744269807},{"type":"v1","addr":"172.21.15.19:6813","nonce":1744269807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":1744269807},{"type":"v1","addr":"172.21.15.19:6817","nonce":1744269807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":1744269807},{"type":"v1","addr":"172.21.15.19:6815","nonce":1744269807}]},"public_addr":"172.21.15.19:6811/1744269807","cluster_addr":"172.21.15.19:6813/1744269807","heartbeat_back_addr":"172.21.15.19:6817/1744269807","heartbeat_front_addr":"172.21.15.19:6815/1744269807","state":["exists","up"]},{"osd":2,"uuid":"b59081e7-53c6-46df-b66e-44f5cf2a79d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":879092529},{"type":"v1","addr":"172.21.15.19:6819","nonce":879092529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":879092529},{"type":"v1","addr":"172.21.15.19:6821","nonce":879092529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":879092529},{"type":"v1","addr":"172.21.15.19:6825","nonce":879092529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":879092529},{"type":"v1","addr":"172.21.15.19:6823","nonce":879092529}]},"public_addr":"172.21.15.19:6819/879092529","cluster_addr":"172.21.15.19:6821/879092529","heartbeat_back_addr":"172.21.15.19:6825/879092529","heartbeat_front_addr":"172.21.15.19:6823/879092529","state":["exists","up"]},{"osd":3,"uuid":"1031e82e-75ee-484d-aa7a-2ee1d697bdc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":3406505853},{"type":"v1","addr":"172.21.15.19:6827","nonce":3406505853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":3406505853},{"type":"v1","addr":"172.21.15.19:6829","nonce":3406505853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":3406505853},{"type":"v1","addr":"172.21.15.19:6833","nonce":3406505853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":3406505853},{"type":"v1","addr":"172.21.15.19:6831","nonce":3406505853}]},"public_addr":"172.21.15.19:6827/3406505853","cluster_addr":"172.21.15.19:6829/3406505853","heartbeat_back_addr":"172.21.15.19:6833/3406505853","heartbeat_front_addr":"172.21.15.19:6831/3406505853","state":["exists","up"]},{"osd":4,"uuid":"12b11e94-3bef-4235-925d-083e55507ba5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1930906476},{"type":"v1","addr":"172.21.15.69:6801","nonce":1930906476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1930906476},{"type":"v1","addr":"172.21.15.69:6803","nonce":1930906476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1930906476},{"type":"v1","addr":"172.21.15.69:6807","nonce":1930906476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1930906476},{"type":"v1","addr":"172.21.15.69:6805","nonce":1930906476}]},"public_addr":"172.21.15.69:6801/1930906476","cluster_addr":"172.21.15.69:6803/1930906476","heartbeat_back_addr":"172.21.15.69:6807/1930906476","heartbeat_front_addr":"172.21.15.69:6805/1930906476","state":["exists","up"]},{"osd":5,"uuid":"87ffd30b-283e-4da7-8d79-e4ab62c503fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":3561881892},{"type":"v1","addr":"172.21.15.69:6809","nonce":3561881892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3561881892},{"type":"v1","addr":"172.21.15.69:6811","nonce":3561881892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3561881892},{"type":"v1","addr":"172.21.15.69:6815","nonce":3561881892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3561881892},{"type":"v1","addr":"172.21.15.69:6813","nonce":3561881892}]},"public_addr":"172.21.15.69:6809/3561881892","cluster_addr":"172.21.15.69:6811/3561881892","heartbeat_back_addr":"172.21.15.69:6815/3561881892","heartbeat_front_addr":"172.21.15.69:6813/3561881892","state":["exists","up"]},{"osd":6,"uuid":"ad320d11-c1b3-4dc8-8e60-6fe1bbf7b377","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1173599153},{"type":"v1","addr":"172.21.15.69:6817","nonce":1173599153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1173599153},{"type":"v1","addr":"172.21.15.69:6819","nonce":1173599153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":1173599153},{"type":"v1","addr":"172.21.15.69:6823","nonce":1173599153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":1173599153},{"type":"v1","addr":"172.21.15.69:6821","nonce":1173599153}]},"public_addr":"172.21.15.69:6817/1173599153","cluster_addr":"172.21.15.69:6819/1173599153","heartbeat_back_addr":"172.21.15.69:6823/1173599153","heartbeat_front_addr":"172.21.15.69:6821/1173599153","state":["exists","up"]},{"osd":7,"uuid":"1882c969-5f76-4d9f-ba58-73fd558d0d0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":2299762713},{"type":"v1","addr":"172.21.15.69:6825","nonce":2299762713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":2299762713},{"type":"v1","addr":"172.21.15.69:6827","nonce":2299762713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2299762713},{"type":"v1","addr":"172.21.15.69:6831","nonce":2299762713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":2299762713},{"type":"v1","addr":"172.21.15.69:6829","nonce":2299762713}]},"public_addr":"172.21.15.69:6825/2299762713","cluster_addr":"172.21.15.69:6827/2299762713","heartbeat_back_addr":"172.21.15.69:6831/2299762713","heartbeat_front_addr":"172.21.15.69:6829/2299762713","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:48:29.818218+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:49:02.640823+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:49:28.738825+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:49:54.849692+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:50:16.506248+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:50:39.958875+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:51:04.201784+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:51:29.186801+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/3333157499":"2024-03-27T01:47:18.060767+0000","172.21.15.19:0/3485031281":"2024-03-27T01:47:18.060767+0000","172.21.15.19:0/958839096":"2024-03-27T01:47:18.060767+0000","172.21.15.19:0/2658771683":"2024-03-27T01:46:23.501942+0000","172.21.15.19:0/622698413":"2024-03-27T01:46:23.501942+0000","172.21.15.19:6800/2285111467":"2024-03-27T01:47:18.060767+0000","172.21.15.19:6800/771609471":"2024-03-27T01:46:23.501942+0000","172.21.15.19:0/3007239008":"2024-03-27T01:46:23.501942+0000","172.21.15.19:0/1610114816":"2024-03-27T01:45:54.671764+0000","172.21.15.19:6801/771609471":"2024-03-27T01:46:23.501942+0000","172.21.15.19:6800/101591818":"2024-03-27T01:45:54.671764+0000","172.21.15.19:6801/101591818":"2024-03-27T01:45:54.671764+0000","172.21.15.19:6801/2285111467":"2024-03-27T01:47:18.060767+0000","172.21.15.19:0/3047936116":"2024-03-27T01:45:54.671764+0000","172.21.15.19:0/1016723143":"2024-03-27T01:45:54.671764+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T01:51:49.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:48 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/3519577238' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T01:51:49.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:48 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/3519577238' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T01:51:49.340 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-26T01:51:49.341 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-26T01:51:49.341 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-26T01:51:49.342 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-26T01:51:49.342 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-26T01:51:49.342 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-26T01:51:49.343 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-26T01:51:49.343 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-26T01:51:49.778 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/mon.smithi019/config 2024-03-26T01:51:49.779 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/mon.smithi019/config 2024-03-26T01:51:49.788 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/mon.smithi019/config 2024-03-26T01:51:49.793 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/mon.smithi019/config 2024-03-26T01:51:49.793 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/mon.smithi019/config 2024-03-26T01:51:49.794 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/mon.smithi019/config 2024-03-26T01:51:49.798 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/mon.smithi019/config 2024-03-26T01:51:49.798 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/mon.smithi019/config 2024-03-26T01:51:50.009 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:49 smithi019 ceph-mon[24816]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:51:50.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:49 smithi069 ceph-mon[27071]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:51:51.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:51 smithi069 ceph-mon[27071]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:51:52.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:51 smithi019 ceph-mon[24816]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:51:53.885 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:53 smithi019 ceph-mon[24816]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:51:54.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:53 smithi069 ceph-mon[27071]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:51:55.881 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:55 smithi019 ceph-mon[24816]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:51:56.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:55 smithi069 ceph-mon[27071]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:51:58.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:57 smithi019 ceph-mon[24816]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:51:58.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:57 smithi069 ceph-mon[27071]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:51:59.004 INFO:teuthology.orchestra.run.smithi019.stdout:133143986198 2024-03-26T01:51:59.004 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-26T01:51:59.887 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:51:59 smithi019 ceph-mon[24816]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:00.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:51:59 smithi069 ceph-mon[27071]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:01.595 INFO:teuthology.orchestra.run.smithi019.stdout:236223201288 2024-03-26T01:52:01.596 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-26T01:52:01.834 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:01 smithi019 ceph-mon[24816]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:01.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:01 smithi069 ceph-mon[27071]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:02.209 INFO:teuthology.orchestra.run.smithi019.stdout:81604378656 2024-03-26T01:52:02.209 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-26T01:52:03.137 INFO:teuthology.orchestra.run.smithi019.stdout:38654705709 2024-03-26T01:52:03.137 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-26T01:52:03.971 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:03 smithi019 ceph-mon[24816]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:04.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:03 smithi069 ceph-mon[27071]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:05.441 INFO:teuthology.orchestra.run.smithi019.stdout:201863462926 2024-03-26T01:52:05.442 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-26T01:52:05.911 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:05 smithi069 ceph-mon[27071]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:05.955 INFO:teuthology.orchestra.run.smithi019.stdout:167503724563 2024-03-26T01:52:05.955 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-26T01:52:06.094 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:05 smithi019 ceph-mon[24816]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:06.492 INFO:teuthology.orchestra.run.smithi019.stdout:111669149724 2024-03-26T01:52:06.493 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-26T01:52:06.914 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/mon.smithi019/config 2024-03-26T01:52:06.915 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/mon.smithi019/config 2024-03-26T01:52:06.916 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/mon.smithi019/config 2024-03-26T01:52:06.916 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/mon.smithi019/config 2024-03-26T01:52:06.917 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/mon.smithi019/config 2024-03-26T01:52:06.920 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/mon.smithi019/config 2024-03-26T01:52:06.952 INFO:teuthology.orchestra.run.smithi019.stdout:60129542183 2024-03-26T01:52:06.952 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-26T01:52:08.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:07 smithi069 ceph-mon[27071]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:08.123 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:07 smithi019 ceph-mon[24816]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:10.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:09 smithi019 ceph-mon[24816]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:10.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:09 smithi069 ceph-mon[27071]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:12.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:11 smithi069 ceph-mon[27071]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:12.127 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:11 smithi019 ceph-mon[24816]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:13.012 INFO:teuthology.orchestra.run.smithi019.stdout:236223201290 2024-03-26T01:52:13.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:12 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/3558539189' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T01:52:13.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:12 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/3558539189' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T01:52:14.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:13 smithi069 ceph-mon[27071]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:14.136 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:13 smithi019 ceph-mon[24816]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:14.222 INFO:teuthology.orchestra.run.smithi019.stdout:38654705712 2024-03-26T01:52:14.426 INFO:teuthology.orchestra.run.smithi019.stdout:133143986201 2024-03-26T01:52:15.100 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:14 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/4059671908' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T01:52:15.100 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:14 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/4284430730' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T01:52:15.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:14 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/4059671908' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T01:52:15.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:14 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/4284430730' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T01:52:15.128 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986198 got 133143986201 for osd.4 2024-03-26T01:52:15.129 DEBUG:teuthology.parallel:result is None 2024-03-26T01:52:15.572 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201288 got 236223201290 for osd.7 2024-03-26T01:52:15.573 DEBUG:teuthology.parallel:result is None 2024-03-26T01:52:15.780 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/mon.smithi019/config 2024-03-26T01:52:16.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:15 smithi069 ceph-mon[27071]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:16.141 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:15 smithi019 ceph-mon[24816]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:16.168 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705709 got 38654705712 for osd.0 2024-03-26T01:52:16.168 DEBUG:teuthology.parallel:result is None 2024-03-26T01:52:16.582 INFO:teuthology.orchestra.run.smithi019.stdout:201863462928 2024-03-26T01:52:16.850 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/mon.smithi019/config 2024-03-26T01:52:17.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:16 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/3711118710' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T01:52:17.176 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:16 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/3711118710' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T01:52:18.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:17 smithi019 ceph-mon[24816]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:18.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:17 smithi069 ceph-mon[27071]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:18.381 INFO:teuthology.orchestra.run.smithi019.stdout:167503724566 2024-03-26T01:52:19.143 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:18 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/1830476405' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T01:52:19.143 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:18 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/trash_purge_schedule"}]: dispatch 2024-03-26T01:52:19.143 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:18 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:52:19.165 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724563 got 167503724566 for osd.5 2024-03-26T01:52:19.165 DEBUG:teuthology.parallel:result is None 2024-03-26T01:52:19.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:18 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/1830476405' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T01:52:19.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:18 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/trash_purge_schedule"}]: dispatch 2024-03-26T01:52:19.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:18 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:52:19.520 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462926 got 201863462928 for osd.6 2024-03-26T01:52:19.520 DEBUG:teuthology.parallel:result is None 2024-03-26T01:52:20.066 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:19 smithi019 ceph-mon[24816]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:20.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:19 smithi069 ceph-mon[27071]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:20.648 INFO:teuthology.orchestra.run.smithi019.stdout:60129542186 2024-03-26T01:52:21.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:20 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/1485960454' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T01:52:21.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:20 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/1485960454' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T01:52:21.801 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542183 got 60129542186 for osd.1 2024-03-26T01:52:21.801 DEBUG:teuthology.parallel:result is None 2024-03-26T01:52:22.062 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:21 smithi019 ceph-mon[24816]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:22.262 INFO:teuthology.orchestra.run.smithi019.stdout:81604378660 2024-03-26T01:52:22.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:21 smithi069 ceph-mon[27071]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:22.790 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378656 got 81604378660 for osd.2 2024-03-26T01:52:22.790 DEBUG:teuthology.parallel:result is None 2024-03-26T01:52:23.238 INFO:teuthology.orchestra.run.smithi019.stdout:111669149727 2024-03-26T01:52:23.239 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:22 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/2931021187' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T01:52:23.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:22 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/2931021187' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T01:52:23.896 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:23 smithi019 ceph-mon[24816]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:23.897 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:23 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/2168432845' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T01:52:23.920 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149724 got 111669149727 for osd.3 2024-03-26T01:52:23.920 DEBUG:teuthology.parallel:result is None 2024-03-26T01:52:23.920 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-26T01:52:23.921 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T01:52:24.141 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/mon.smithi019/config 2024-03-26T01:52:24.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:23 smithi069 ceph-mon[27071]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:24.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:23 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/2168432845' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T01:52:25.884 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-26T01:52:25.886 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-03-26T01:52:26.260 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:25 smithi019 ceph-mon[24816]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:26.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:25 smithi069 ceph-mon[27071]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:26.520 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-03-26T01:52:24.747762+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48932,"kb_used_data":6308,"kb_used_omap":0,"kb_used_meta":42560,"kb_avail":749912284,"statfs":{"total":767960285184,"available":767910178816,"internally_reserved":0,"allocated":6459392,"data_stored":3547207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43581440},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002261"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":226,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-26T01:51:35.098317+0000","last_change":"2024-03-26T01:51:35.098317+0000","last_active":"2024-03-26T01:51:35.098317+0000","last_peered":"2024-03-26T01:51:35.098317+0000","last_clean":"2024-03-26T01:51:35.098317+0000","last_became_active":"2024-03-26T01:51:35.097731+0000","last_became_peered":"2024-03-26T01:51:35.097731+0000","last_unstale":"2024-03-26T01:51:35.098317+0000","last_undegraded":"2024-03-26T01:51:35.098317+0000","last_fullsized":"2024-03-26T01:51:35.098317+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:49:31.778162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:49:31.778162+0000","last_clean_scrub_stamp":"2024-03-26T01:49:31.778162+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:40:23.497400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2662400,"data_stored":2644640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201293,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6052,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739100,"statfs":{"total":95995035648,"available":95988838400,"internally_reserved":0,"allocated":1142784,"data_stored":774161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]}]},{"osd":6,"up_from":47,"seq":201863462930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6176,"kb_used_data":1112,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738976,"statfs":{"total":95995035648,"available":95988711424,"internally_reserved":0,"allocated":1138688,"data_stored":773838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 01:52:06 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.60999999999999999}]},{"osd":1,"last update":"Tue Mar 26 01:52:06 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.417,"15min":1.417},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":16.527999999999999,"5min":16.527999999999999,"15min":16.527999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.63300000000000001}]},{"osd":2,"last update":"Tue Mar 26 01:52:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.68500000000000005}]},{"osd":3,"last update":"Tue Mar 26 01:52:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":16.568999999999999,"5min":16.568999999999999,"15min":16.568999999999999},"last":0.624},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.69899999999999995}]},{"osd":4,"last update":"Tue Mar 26 01:52:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.497}]},{"osd":5,"last update":"Tue Mar 26 01:52:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.58299999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]}]},{"osd":1,"up_from":14,"seq":60129542187,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6356,"kb_used_data":588,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738796,"statfs":{"total":95995035648,"available":95988527104,"internally_reserved":0,"allocated":602112,"data_stored":244703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 01:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.35599999999999998,"15min":0.33400000000000002},"min":{"1min":0.28100000000000003,"5min":0.191,"15min":0.191},"max":{"1min":0.51900000000000002,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.38600000000000001,"15min":0.35799999999999998},"min":{"1min":0.35799999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.57799999999999996,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.33200000000000002}]},{"osd":2,"last update":"Tue Mar 26 01:51:33 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.43099999999999999},"min":{"1min":0.26800000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.55800000000000005,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45600000000000002,"15min":0.45700000000000002},"min":{"1min":0.316,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.60099999999999998,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.40000000000000002}]},{"osd":3,"last update":"Tue Mar 26 01:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.45000000000000001,"15min":0.44800000000000001},"min":{"1min":0.315,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.59999999999999998,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47699999999999998,"15min":0.47499999999999998},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.624,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.39100000000000001}]},{"osd":4,"last update":"Tue Mar 26 01:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.44400000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.77000000000000002,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.56899999999999995}]},{"osd":5,"last update":"Tue Mar 26 01:51:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.48899999999999999}]},{"osd":6,"last update":"Tue Mar 26 01:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.46899999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705714,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6884,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738268,"statfs":{"total":95995035648,"available":95987986432,"internally_reserved":0,"allocated":1142784,"data_stored":774161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Mar 26 01:52:15 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.35999999999999999,"15min":0.34300000000000003},"min":{"1min":0.224,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.63900000000000001,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.313},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.35399999999999998,"15min":0.33600000000000002},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.59699999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.30299999999999999}]},{"osd":2,"last update":"Tue Mar 26 01:51:36 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.36799999999999999,"15min":0.36399999999999999},"min":{"1min":0.27800000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.57999999999999996,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.25},{"interface":"front","average":{"1min":0.432,"5min":0.40300000000000002,"15min":0.39800000000000002},"min":{"1min":0.29799999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.38800000000000001}]},{"osd":3,"last update":"Tue Mar 26 01:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.443,"15min":0.44500000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.58199999999999996,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46400000000000002,"15min":0.46600000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.34200000000000003}]},{"osd":4,"last update":"Tue Mar 26 01:52:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.621,"15min":0.61699999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.58999999999999997,"15min":0.58499999999999996},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.70099999999999996}]},{"osd":5,"last update":"Tue Mar 26 01:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.67900000000000005}]},{"osd":6,"last update":"Tue Mar 26 01:52:07 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.56499999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]}]},{"osd":2,"up_from":19,"seq":81604378661,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6300,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738852,"statfs":{"total":95995035648,"available":95988584448,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 01:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.36899999999999999,"15min":0.35799999999999998},"min":{"1min":0.28499999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.57199999999999995,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.44,"5min":0.36799999999999999,"15min":0.35599999999999998},"min":{"1min":0.30199999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.53700000000000003}]},{"osd":1,"last update":"Tue Mar 26 01:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.38700000000000001,"15min":0.378},"min":{"1min":0.33900000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.56299999999999994,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.436,"5min":0.377,"15min":0.36699999999999999},"min":{"1min":0.28999999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.628,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.43099999999999999}]},{"osd":3,"last update":"Tue Mar 26 01:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.443,"15min":0.443},"min":{"1min":0.32900000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.48399999999999999,"15min":0.47899999999999998},"min":{"1min":0.32400000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.69399999999999995,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.56799999999999995}]},{"osd":4,"last update":"Tue Mar 26 01:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.66900000000000004}]},{"osd":5,"last update":"Tue Mar 26 01:51:47 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.754}]},{"osd":6,"last update":"Tue Mar 26 01:52:11 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":4.0140000000000002,"5min":4.0140000000000002,"15min":4.0140000000000002},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.70899999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149728,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5724,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739428,"statfs":{"total":95995035648,"available":95989174272,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 01:52:06 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.38100000000000001,"15min":0.38600000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.48499999999999999,"5min":0.621,"15min":0.621},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.40699999999999997,"15min":0.40999999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.432}]},{"osd":1,"last update":"Tue Mar 26 01:52:06 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.437,"15min":0.441},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.55800000000000005,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.41299999999999998,"15min":0.41499999999999998},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.41199999999999998}]},{"osd":2,"last update":"Tue Mar 26 01:52:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.495,"15min":0.503},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.61799999999999999,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.45400000000000001,"15min":0.45800000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.378}]},{"osd":4,"last update":"Tue Mar 26 01:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.65200000000000002}]},{"osd":5,"last update":"Tue Mar 26 01:51:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.72799999999999998}]},{"osd":6,"last update":"Tue Mar 26 01:52:06 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.70699999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]}]},{"osd":4,"up_from":31,"seq":133143986204,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5784,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739368,"statfs":{"total":95995035648,"available":95989112832,"internally_reserved":0,"allocated":606208,"data_stored":244968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 01:52:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.53100000000000003,"15min":0.52500000000000002},"min":{"1min":0.34499999999999997,"5min":0.252,"15min":0.252},"max":{"1min":0.81200000000000006,"5min":0.871,"15min":0.871},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.55700000000000005,"15min":0.55100000000000005},"min":{"1min":0.38600000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.81999999999999995,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.57499999999999996}]},{"osd":1,"last update":"Tue Mar 26 01:52:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.501,"15min":0.497},"min":{"1min":0.37,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.80000000000000004,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.50900000000000001,"15min":0.499},"min":{"1min":0.40899999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.70599999999999996,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.40899999999999997}]},{"osd":2,"last update":"Tue Mar 26 01:52:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.51300000000000001,"15min":0.503},"min":{"1min":0.374,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.77600000000000002,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.374},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.52300000000000002,"15min":0.51500000000000001},"min":{"1min":0.41699999999999998,"5min":0.378,"15min":0.378},"max":{"1min":0.72099999999999997,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.56399999999999995}]},{"osd":3,"last update":"Tue Mar 26 01:52:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.53600000000000003,"15min":0.53200000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.74299999999999999,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.52300000000000002,"15min":0.51300000000000001},"min":{"1min":0.47699999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.502}]},{"osd":5,"last update":"Tue Mar 26 01:51:45 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.53600000000000003}]},{"osd":6,"last update":"Tue Mar 26 01:52:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.54300000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]}]},{"osd":5,"up_from":39,"seq":167503724568,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5656,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739496,"statfs":{"total":95995035648,"available":95989243904,"internally_reserved":0,"allocated":606208,"data_stored":244910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 01:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.42799999999999999}]},{"osd":1,"last update":"Tue Mar 26 01:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.47599999999999998}]},{"osd":2,"last update":"Tue Mar 26 01:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.60899999999999999}]},{"osd":3,"last update":"Tue Mar 26 01:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.65100000000000002}]},{"osd":4,"last update":"Tue Mar 26 01:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.70599999999999996}]},{"osd":6,"last update":"Tue Mar 26 01:52:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.73299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T01:52:26.521 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T01:52:26.741 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/mon.smithi019/config 2024-03-26T01:52:27.171 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:26 smithi019 ceph-mon[24816]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T01:52:27.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:26 smithi069 ceph-mon[27071]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T01:52:28.174 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:27 smithi019 ceph-mon[24816]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:28.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:27 smithi069 ceph-mon[27071]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:28.554 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-26T01:52:28.556 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-03-26T01:52:29.340 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-03-26T01:52:26.748235+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48932,"kb_used_data":6308,"kb_used_omap":0,"kb_used_meta":42560,"kb_avail":749912284,"statfs":{"total":767960285184,"available":767910178816,"internally_reserved":0,"allocated":6459392,"data_stored":3547207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43581440},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002252"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":226,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-26T01:51:35.098317+0000","last_change":"2024-03-26T01:51:35.098317+0000","last_active":"2024-03-26T01:51:35.098317+0000","last_peered":"2024-03-26T01:51:35.098317+0000","last_clean":"2024-03-26T01:51:35.098317+0000","last_became_active":"2024-03-26T01:51:35.097731+0000","last_became_peered":"2024-03-26T01:51:35.097731+0000","last_unstale":"2024-03-26T01:51:35.098317+0000","last_undegraded":"2024-03-26T01:51:35.098317+0000","last_fullsized":"2024-03-26T01:51:35.098317+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:49:31.778162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:49:31.778162+0000","last_clean_scrub_stamp":"2024-03-26T01:49:31.778162+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:40:23.497400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2662400,"data_stored":2644640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201294,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6052,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739100,"statfs":{"total":95995035648,"available":95988838400,"internally_reserved":0,"allocated":1142784,"data_stored":774161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]}]},{"osd":6,"up_from":47,"seq":201863462931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6176,"kb_used_data":1112,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738976,"statfs":{"total":95995035648,"available":95988711424,"internally_reserved":0,"allocated":1138688,"data_stored":773838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 01:52:06 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.65000000000000002}]},{"osd":1,"last update":"Tue Mar 26 01:52:06 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.417,"15min":1.417},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":16.527999999999999,"5min":16.527999999999999,"15min":16.527999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.57999999999999996}]},{"osd":2,"last update":"Tue Mar 26 01:52:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.628},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.56999999999999995}]},{"osd":3,"last update":"Tue Mar 26 01:52:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":16.568999999999999,"5min":16.568999999999999,"15min":16.568999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.53400000000000003}]},{"osd":4,"last update":"Tue Mar 26 01:52:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.51500000000000001}]},{"osd":5,"last update":"Tue Mar 26 01:52:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.42999999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542188,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6356,"kb_used_data":588,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738796,"statfs":{"total":95995035648,"available":95988527104,"internally_reserved":0,"allocated":602112,"data_stored":244703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 01:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.35599999999999998,"15min":0.33400000000000002},"min":{"1min":0.28100000000000003,"5min":0.191,"15min":0.191},"max":{"1min":0.51900000000000002,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.38600000000000001,"15min":0.35799999999999998},"min":{"1min":0.35799999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.57799999999999996,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.33200000000000002}]},{"osd":2,"last update":"Tue Mar 26 01:51:33 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.43099999999999999},"min":{"1min":0.26800000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.55800000000000005,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45600000000000002,"15min":0.45700000000000002},"min":{"1min":0.316,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.60099999999999998,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.40000000000000002}]},{"osd":3,"last update":"Tue Mar 26 01:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.45000000000000001,"15min":0.44800000000000001},"min":{"1min":0.315,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.59999999999999998,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47699999999999998,"15min":0.47499999999999998},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.624,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.39100000000000001}]},{"osd":4,"last update":"Tue Mar 26 01:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.44400000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.77000000000000002,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.56899999999999995}]},{"osd":5,"last update":"Tue Mar 26 01:51:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.48899999999999999}]},{"osd":6,"last update":"Tue Mar 26 01:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.46899999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705714,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6884,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738268,"statfs":{"total":95995035648,"available":95987986432,"internally_reserved":0,"allocated":1142784,"data_stored":774161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Mar 26 01:52:15 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.35999999999999999,"15min":0.34300000000000003},"min":{"1min":0.224,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.63900000000000001,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.313},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.35399999999999998,"15min":0.33600000000000002},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.59699999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.30299999999999999}]},{"osd":2,"last update":"Tue Mar 26 01:51:36 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.36799999999999999,"15min":0.36399999999999999},"min":{"1min":0.27800000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.57999999999999996,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.25},{"interface":"front","average":{"1min":0.432,"5min":0.40300000000000002,"15min":0.39800000000000002},"min":{"1min":0.29799999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.38800000000000001}]},{"osd":3,"last update":"Tue Mar 26 01:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.443,"15min":0.44500000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.58199999999999996,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46400000000000002,"15min":0.46600000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.34200000000000003}]},{"osd":4,"last update":"Tue Mar 26 01:52:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.621,"15min":0.61699999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.58999999999999997,"15min":0.58499999999999996},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.70099999999999996}]},{"osd":5,"last update":"Tue Mar 26 01:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.67900000000000005}]},{"osd":6,"last update":"Tue Mar 26 01:52:07 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.56499999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]}]},{"osd":2,"up_from":19,"seq":81604378661,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6300,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738852,"statfs":{"total":95995035648,"available":95988584448,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 01:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.36899999999999999,"15min":0.35799999999999998},"min":{"1min":0.28499999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.57199999999999995,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.44,"5min":0.36799999999999999,"15min":0.35599999999999998},"min":{"1min":0.30199999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.53700000000000003}]},{"osd":1,"last update":"Tue Mar 26 01:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.38700000000000001,"15min":0.378},"min":{"1min":0.33900000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.56299999999999994,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.436,"5min":0.377,"15min":0.36699999999999999},"min":{"1min":0.28999999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.628,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.43099999999999999}]},{"osd":3,"last update":"Tue Mar 26 01:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.443,"15min":0.443},"min":{"1min":0.32900000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.48399999999999999,"15min":0.47899999999999998},"min":{"1min":0.32400000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.69399999999999995,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.56799999999999995}]},{"osd":4,"last update":"Tue Mar 26 01:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.66900000000000004}]},{"osd":5,"last update":"Tue Mar 26 01:51:47 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.754}]},{"osd":6,"last update":"Tue Mar 26 01:52:11 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":4.0140000000000002,"5min":4.0140000000000002,"15min":4.0140000000000002},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.70899999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149728,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5724,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739428,"statfs":{"total":95995035648,"available":95989174272,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 01:52:06 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.38100000000000001,"15min":0.38600000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.48499999999999999,"5min":0.621,"15min":0.621},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.40699999999999997,"15min":0.40999999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.432}]},{"osd":1,"last update":"Tue Mar 26 01:52:06 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.437,"15min":0.441},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.55800000000000005,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.41299999999999998,"15min":0.41499999999999998},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.41199999999999998}]},{"osd":2,"last update":"Tue Mar 26 01:52:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.495,"15min":0.503},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.61799999999999999,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.45400000000000001,"15min":0.45800000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.378}]},{"osd":4,"last update":"Tue Mar 26 01:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.65200000000000002}]},{"osd":5,"last update":"Tue Mar 26 01:51:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.72799999999999998}]},{"osd":6,"last update":"Tue Mar 26 01:52:06 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.70699999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]}]},{"osd":4,"up_from":31,"seq":133143986204,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5784,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739368,"statfs":{"total":95995035648,"available":95989112832,"internally_reserved":0,"allocated":606208,"data_stored":244968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 01:52:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.53100000000000003,"15min":0.52500000000000002},"min":{"1min":0.34499999999999997,"5min":0.252,"15min":0.252},"max":{"1min":0.81200000000000006,"5min":0.871,"15min":0.871},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.55700000000000005,"15min":0.55100000000000005},"min":{"1min":0.38600000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.81999999999999995,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.57499999999999996}]},{"osd":1,"last update":"Tue Mar 26 01:52:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.501,"15min":0.497},"min":{"1min":0.37,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.80000000000000004,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.50900000000000001,"15min":0.499},"min":{"1min":0.40899999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.70599999999999996,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.40899999999999997}]},{"osd":2,"last update":"Tue Mar 26 01:52:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.51300000000000001,"15min":0.503},"min":{"1min":0.374,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.77600000000000002,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.374},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.52300000000000002,"15min":0.51500000000000001},"min":{"1min":0.41699999999999998,"5min":0.378,"15min":0.378},"max":{"1min":0.72099999999999997,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.56399999999999995}]},{"osd":3,"last update":"Tue Mar 26 01:52:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.53600000000000003,"15min":0.53200000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.74299999999999999,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.52300000000000002,"15min":0.51300000000000001},"min":{"1min":0.47699999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.502}]},{"osd":5,"last update":"Tue Mar 26 01:51:45 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.53600000000000003}]},{"osd":6,"last update":"Tue Mar 26 01:52:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.54300000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]}]},{"osd":5,"up_from":39,"seq":167503724568,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5656,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739496,"statfs":{"total":95995035648,"available":95989243904,"internally_reserved":0,"allocated":606208,"data_stored":244910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 01:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.42799999999999999}]},{"osd":1,"last update":"Tue Mar 26 01:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.47599999999999998}]},{"osd":2,"last update":"Tue Mar 26 01:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.60899999999999999}]},{"osd":3,"last update":"Tue Mar 26 01:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.65100000000000002}]},{"osd":4,"last update":"Tue Mar 26 01:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.70599999999999996}]},{"osd":6,"last update":"Tue Mar 26 01:52:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.73299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T01:52:29.342 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-26T01:52:29.342 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-26T01:52:29.342 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-26T01:52:29.342 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph health --format=json 2024-03-26T01:52:29.564 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/mon.smithi019/config 2024-03-26T01:52:29.976 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:29 smithi019 ceph-mon[24816]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T01:52:29.976 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:29 smithi019 ceph-mon[24816]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:30.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:29 smithi069 ceph-mon[27071]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T01:52:30.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:29 smithi069 ceph-mon[27071]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:31.212 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-26T01:52:31.212 INFO:teuthology.orchestra.run.smithi019.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-26T01:52:31.787 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-26T01:52:31.787 INFO:tasks.cephadm:Setup complete, yielding 2024-03-26T01:52:31.788 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-26T01:52:31.797 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi019.front.sepia.ceph.com 2024-03-26T01:52:31.798 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- bash -c 'ceph orch status' 2024-03-26T01:52:32.229 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:31 smithi019 ceph-mon[24816]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:32.229 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:31 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/207756699' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T01:52:32.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:31 smithi069 ceph-mon[27071]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:32.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:31 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/207756699' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T01:52:33.783 INFO:teuthology.orchestra.run.smithi019.stdout:Backend: cephadm 2024-03-26T01:52:33.783 INFO:teuthology.orchestra.run.smithi019.stdout:Available: Yes 2024-03-26T01:52:33.783 INFO:teuthology.orchestra.run.smithi019.stdout:Paused: No 2024-03-26T01:52:34.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:33 smithi019 ceph-mon[24816]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:34.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:33 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:52:34.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:33 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:52:34.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:33 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:52:34.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:33 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:52:34.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:33 smithi069 ceph-mon[27071]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:34.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:33 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:52:34.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:33 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:52:34.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:33 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:52:34.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:33 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:52:34.393 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-26T01:52:34.979 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:34 smithi019 ceph-mon[24816]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:52:35.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:34 smithi069 ceph-mon[27071]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:52:35.960 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:35 smithi019 ceph-mon[24816]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:36.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:35 smithi069 ceph-mon[27071]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:36.349 INFO:teuthology.orchestra.run.smithi019.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T01:52:36.349 INFO:teuthology.orchestra.run.smithi019.stdout:alertmanager.smithi019 smithi019 *:9093,9094 running (4m) 2m ago 5m 15.5M - ba2b418f427c b72e14ab8874 2024-03-26T01:52:36.349 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi019 smithi019 running (5m) 2m ago 5m 6405k - 17.2.0 e1d6a67b021e 713db693d5c1 2024-03-26T01:52:36.349 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi069 smithi069 running (5m) 66s ago 5m 6405k - 17.2.0 e1d6a67b021e a87d27e2b74b 2024-03-26T01:52:36.349 INFO:teuthology.orchestra.run.smithi019.stdout:grafana.smithi019 smithi019 *:3000 running (4m) 2m ago 5m 43.5M - 8.3.5 dad864ee21e9 81e9f452f143 2024-03-26T01:52:36.349 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi019.ixuvdm smithi019 *:9283 running (7m) 2m ago 7m 447M - 17.2.0 e1d6a67b021e 1d13a7db0753 2024-03-26T01:52:36.349 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi069.wpbewu smithi069 *:8443,9283 running (5m) 66s ago 5m 411M - 17.2.0 e1d6a67b021e 0a4bd07111bb 2024-03-26T01:52:36.349 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi019 smithi019 running (7m) 2m ago 7m 41.2M 2048M 17.2.0 e1d6a67b021e b6908b497829 2024-03-26T01:52:36.349 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi069 smithi069 running (4m) 66s ago 4m 41.8M 2048M 17.2.0 e1d6a67b021e 2d51b7b7bb32 2024-03-26T01:52:36.350 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi019 smithi019 *:9100 running (5m) 2m ago 5m 17.3M - 1dbe0e931976 8094999d4d38 2024-03-26T01:52:36.350 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi069 smithi069 *:9100 running (4m) 66s ago 4m 17.0M - 1dbe0e931976 46e719c0de82 2024-03-26T01:52:36.350 INFO:teuthology.orchestra.run.smithi019.stdout:osd.0 smithi019 running (4m) 2m ago 4m 32.9M 3217M 17.2.0 e1d6a67b021e c1a797230ab6 2024-03-26T01:52:36.350 INFO:teuthology.orchestra.run.smithi019.stdout:osd.1 smithi019 running (3m) 2m ago 3m 37.0M 3217M 17.2.0 e1d6a67b021e 31d3489c7188 2024-03-26T01:52:36.350 INFO:teuthology.orchestra.run.smithi019.stdout:osd.2 smithi019 running (3m) 2m ago 3m 32.9M 3217M 17.2.0 e1d6a67b021e f549115e7685 2024-03-26T01:52:36.350 INFO:teuthology.orchestra.run.smithi019.stdout:osd.3 smithi019 running (2m) 2m ago 2m 10.1M 3217M 17.2.0 e1d6a67b021e 154af68d03e1 2024-03-26T01:52:36.350 INFO:teuthology.orchestra.run.smithi019.stdout:osd.4 smithi069 running (2m) 66s ago 2m 33.1M 3984M 17.2.0 e1d6a67b021e 4fd5e853f8bf 2024-03-26T01:52:36.350 INFO:teuthology.orchestra.run.smithi019.stdout:osd.5 smithi069 running (2m) 66s ago 2m 33.1M 3984M 17.2.0 e1d6a67b021e 932e401d87c9 2024-03-26T01:52:36.350 INFO:teuthology.orchestra.run.smithi019.stdout:osd.6 smithi069 running (96s) 66s ago 96s 32.6M 3984M 17.2.0 e1d6a67b021e b123f75154c8 2024-03-26T01:52:36.350 INFO:teuthology.orchestra.run.smithi019.stdout:osd.7 smithi069 running (71s) 66s ago 71s 13.0M 3984M 17.2.0 e1d6a67b021e d175fbc9ff8a 2024-03-26T01:52:36.350 INFO:teuthology.orchestra.run.smithi019.stdout:prometheus.smithi019 smithi019 *:9095 running (4m) 2m ago 4m 54.4M - 514e6a882f6e 33689b2df9a5 2024-03-26T01:52:36.922 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-26T01:52:37.918 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:37 smithi019 ceph-mon[24816]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:52:37.918 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:37 smithi019 ceph-mon[24816]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:38.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:37 smithi069 ceph-mon[27071]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:52:38.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:37 smithi069 ceph-mon[27071]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:38.874 INFO:teuthology.orchestra.run.smithi019.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-26T01:52:38.874 INFO:teuthology.orchestra.run.smithi019.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-03-26T01:52:38.874 INFO:teuthology.orchestra.run.smithi019.stdout:crash 2/2 2m ago 6m * 2024-03-26T01:52:38.874 INFO:teuthology.orchestra.run.smithi019.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-03-26T01:52:38.874 INFO:teuthology.orchestra.run.smithi019.stdout:mgr 2/2 2m ago 6m count:2 2024-03-26T01:52:38.874 INFO:teuthology.orchestra.run.smithi019.stdout:mon 2/2 2m ago 5m smithi019:172.21.15.19=smithi019;smithi069:172.21.15.69=smithi069;count:2 2024-03-26T01:52:38.875 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-03-26T01:52:38.875 INFO:teuthology.orchestra.run.smithi019.stdout:osd 8 2m ago - 2024-03-26T01:52:38.875 INFO:teuthology.orchestra.run.smithi019.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-03-26T01:52:39.456 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-26T01:52:40.020 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:39 smithi019 ceph-mon[24816]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:40.020 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:39 smithi019 ceph-mon[24816]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:52:40.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:39 smithi069 ceph-mon[27071]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:40.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:39 smithi069 ceph-mon[27071]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:52:41.455 INFO:teuthology.orchestra.run.smithi019.stdout:HOST ADDR LABELS STATUS 2024-03-26T01:52:41.455 INFO:teuthology.orchestra.run.smithi019.stdout:smithi019 172.21.15.19 2024-03-26T01:52:41.455 INFO:teuthology.orchestra.run.smithi019.stdout:smithi069 172.21.15.69 2024-03-26T01:52:41.455 INFO:teuthology.orchestra.run.smithi019.stdout:2 hosts in cluster 2024-03-26T01:52:41.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:41 smithi069 ceph-mon[27071]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:42.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:41 smithi019 ceph-mon[24816]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:42.087 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-26T01:52:43.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:42 smithi019 ceph-mon[24816]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:52:43.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:42 smithi069 ceph-mon[27071]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:52:43.990 INFO:teuthology.orchestra.run.smithi019.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REJECT REASONS 2024-03-26T01:52:43.990 INFO:teuthology.orchestra.run.smithi019.stdout:smithi019 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5330002G400BGN 400G LVM detected, locked 2024-03-26T01:52:43.990 INFO:teuthology.orchestra.run.smithi019.stdout:smithi069 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400Y0400BGN 400G LVM detected, locked 2024-03-26T01:52:43.991 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:43 smithi019 ceph-mon[24816]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:44.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:43 smithi069 ceph-mon[27071]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:44.656 INFO:teuthology.run_tasks:Running task vip.exec... 2024-03-26T01:52:44.665 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi019.front.sepia.ceph.com 2024-03-26T01:52:44.665 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-03-26T01:52:44.694 INFO:teuthology.orchestra.run.smithi019.stderr:+ systemctl stop nfs-server 2024-03-26T01:52:44.702 INFO:tasks.vip:Running commands on role host.b host ubuntu@smithi069.front.sepia.ceph.com 2024-03-26T01:52:44.702 DEBUG:teuthology.orchestra.run.smithi069:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-03-26T01:52:44.733 INFO:teuthology.orchestra.run.smithi069.stderr:+ systemctl stop nfs-server 2024-03-26T01:52:44.741 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-26T01:52:44.760 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi019.front.sepia.ceph.com 2024-03-26T01:52:44.761 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- bash -c 'ceph fs volume create foofs' 2024-03-26T01:52:45.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:44 smithi019 ceph-mon[24816]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:52:45.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:44 smithi069 ceph-mon[27071]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:52:46.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:45 smithi069 ceph-mon[27071]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:46.128 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:45 smithi019 ceph-mon[24816]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:48.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:47 smithi069 ceph-mon[27071]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:48.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:47 smithi069 ceph-mon[27071]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:52:48.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:47 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-03-26T01:52:48.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:47 smithi019 ceph-mon[24816]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:48.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:47 smithi019 ceph-mon[24816]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:52:48.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:47 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-03-26T01:52:49.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:48 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-03-26T01:52:49.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:48 smithi069 ceph-mon[27071]: osdmap e60: 8 total, 8 up, 8 in 2024-03-26T01:52:49.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:48 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-03-26T01:52:49.197 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:48 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-03-26T01:52:49.197 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:48 smithi019 ceph-mon[24816]: osdmap e60: 8 total, 8 up, 8 in 2024-03-26T01:52:49.197 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:48 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-03-26T01:52:49.197 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:48 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi019[24812]: 2024-03-26T01:52:48.813+0000 7f880c36a700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-26T01:52:49.641 INFO:teuthology.run_tasks:Running task cephadm.wait_for_service... 2024-03-26T01:52:49.650 INFO:tasks.cephadm:Waiting for ceph service mds.foofs to start (timeout 300)... 2024-03-26T01:52:49.650 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph orch ls -f json 2024-03-26T01:52:50.086 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:49 smithi019 ceph-mon[24816]: pgmap v213: 33 pgs: 9 creating+peering, 1 active+clean, 23 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:50.086 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:49 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-03-26T01:52:50.086 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:49 smithi019 ceph-mon[24816]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T01:52:50.086 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:49 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-03-26T01:52:50.086 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:49 smithi019 ceph-mon[24816]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-26T01:52:50.086 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:49 smithi019 ceph-mon[24816]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-26T01:52:50.086 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:49 smithi019 ceph-mon[24816]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T01:52:50.086 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:49 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-03-26T01:52:50.086 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:49 smithi019 ceph-mon[24816]: fsmap foofs:0 2024-03-26T01:52:50.087 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:49 smithi019 ceph-mon[24816]: Saving service mds.foofs spec with placement count:2 2024-03-26T01:52:50.087 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:49 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:52:50.087 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:49 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:52:50.087 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:49 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:52:50.087 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:49 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:52:50.087 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:49 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:52:50.087 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:49 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:52:50.087 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:49 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi019.lmryom", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T01:52:50.087 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:49 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi019.lmryom", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T01:52:50.087 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:49 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:52:50.087 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:49 smithi019 ceph-mon[24816]: Deploying daemon mds.foofs.smithi019.lmryom on smithi019 2024-03-26T01:52:50.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:49 smithi069 ceph-mon[27071]: pgmap v213: 33 pgs: 9 creating+peering, 1 active+clean, 23 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:50.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:49 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-03-26T01:52:50.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:49 smithi069 ceph-mon[27071]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T01:52:50.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:49 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-03-26T01:52:50.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:49 smithi069 ceph-mon[27071]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-26T01:52:50.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:49 smithi069 ceph-mon[27071]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-26T01:52:50.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:49 smithi069 ceph-mon[27071]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T01:52:50.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:49 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-03-26T01:52:50.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:49 smithi069 ceph-mon[27071]: fsmap foofs:0 2024-03-26T01:52:50.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:49 smithi069 ceph-mon[27071]: Saving service mds.foofs spec with placement count:2 2024-03-26T01:52:50.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:49 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:52:50.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:49 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:52:50.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:49 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:52:50.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:49 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:52:50.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:49 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:52:50.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:49 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:52:50.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:49 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi019.lmryom", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T01:52:50.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:49 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi019.lmryom", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T01:52:50.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:49 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:52:50.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:49 smithi069 ceph-mon[27071]: Deploying daemon mds.foofs.smithi019.lmryom on smithi019 2024-03-26T01:52:51.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:50 smithi069 ceph-mon[27071]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T01:52:51.234 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:50 smithi019 ceph-mon[24816]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T01:52:51.825 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:51 smithi019 ceph-mon[24816]: pgmap v217: 65 pgs: 2 creating+activating, 9 creating+peering, 8 active+clean, 46 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:51.825 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:51 smithi019 ceph-mon[24816]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T01:52:52.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:51 smithi069 ceph-mon[27071]: pgmap v217: 65 pgs: 2 creating+activating, 9 creating+peering, 8 active+clean, 46 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:52.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:51 smithi069 ceph-mon[27071]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T01:52:52.847 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-26T01:52:52.847 INFO:teuthology.orchestra.run.smithi019.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-03-26T01:46:13.250596Z", "last_refresh": "2024-03-26T01:49:57.727247Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-03-26T01:46:06.931254Z", "last_refresh": "2024-03-26T01:49:57.727359Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-03-26T01:46:09.898597Z", "last_refresh": "2024-03-26T01:49:57.727430Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-03-26T01:52:48.832653Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-03-26T01:52:48.827688Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-03-26T01:46:05.586775Z", "last_refresh": "2024-03-26T01:49:57.727099Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi019:172.21.15.19=smithi019", "smithi069:172.21.15.69=smithi069"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-03-26T01:46:56.697362Z", "last_refresh": "2024-03-26T01:49:57.726888Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-03-26T01:46:11.134728Z", "last_refresh": "2024-03-26T01:49:57.727496Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-03-26T01:49:57.727624Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-03-26T01:46:08.313914Z", "last_refresh": "2024-03-26T01:49:57.727560Z", "ports": [9095], "running": 1, "size": 1}}] 2024-03-26T01:52:53.579 INFO:tasks.cephadm:mds.foofs has 0/2 2024-03-26T01:52:54.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:53 smithi019 ceph-mon[24816]: pgmap v219: 65 pgs: 2 creating+activating, 9 creating+peering, 39 active+clean, 15 unknown; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:54.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:53 smithi019 ceph-mon[24816]: from='client.14474 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T01:52:54.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:53 smithi069 ceph-mon[27071]: pgmap v219: 65 pgs: 2 creating+activating, 9 creating+peering, 39 active+clean, 15 unknown; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:54.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:53 smithi069 ceph-mon[27071]: from='client.14474 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T01:52:54.580 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph orch ls -f json 2024-03-26T01:52:54.905 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:54 smithi019 ceph-mon[24816]: mds.? [v2:172.21.15.19:6834/3267072550,v1:172.21.15.19:6835/3267072550] up:boot 2024-03-26T01:52:54.905 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:54 smithi019 ceph-mon[24816]: daemon mds.foofs.smithi019.lmryom assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-03-26T01:52:54.905 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:54 smithi019 ceph-mon[24816]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-26T01:52:54.905 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:54 smithi019 ceph-mon[24816]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-26T01:52:54.905 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:54 smithi019 ceph-mon[24816]: Cluster is now healthy 2024-03-26T01:52:54.905 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:54 smithi019 ceph-mon[24816]: fsmap foofs:0 1 up:standby 2024-03-26T01:52:54.905 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:54 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi019.lmryom"}]: dispatch 2024-03-26T01:52:54.906 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:54 smithi019 ceph-mon[24816]: fsmap foofs:1 {0=foofs.smithi019.lmryom=up:creating} 2024-03-26T01:52:54.906 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:54 smithi019 ceph-mon[24816]: daemon mds.foofs.smithi019.lmryom is now active in filesystem foofs as rank 0 2024-03-26T01:52:54.906 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:54 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:52:54.906 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:54 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi069.spgwgo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T01:52:54.906 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:54 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi069.spgwgo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T01:52:54.906 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:54 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:52:54.906 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:54 smithi019 ceph-mon[24816]: Deploying daemon mds.foofs.smithi069.spgwgo on smithi069 2024-03-26T01:52:55.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:54 smithi069 ceph-mon[27071]: mds.? [v2:172.21.15.19:6834/3267072550,v1:172.21.15.19:6835/3267072550] up:boot 2024-03-26T01:52:55.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:54 smithi069 ceph-mon[27071]: daemon mds.foofs.smithi019.lmryom assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-03-26T01:52:55.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:54 smithi069 ceph-mon[27071]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-26T01:52:55.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:54 smithi069 ceph-mon[27071]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-26T01:52:55.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:54 smithi069 ceph-mon[27071]: Cluster is now healthy 2024-03-26T01:52:55.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:54 smithi069 ceph-mon[27071]: fsmap foofs:0 1 up:standby 2024-03-26T01:52:55.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:54 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi019.lmryom"}]: dispatch 2024-03-26T01:52:55.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:54 smithi069 ceph-mon[27071]: fsmap foofs:1 {0=foofs.smithi019.lmryom=up:creating} 2024-03-26T01:52:55.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:54 smithi069 ceph-mon[27071]: daemon mds.foofs.smithi019.lmryom is now active in filesystem foofs as rank 0 2024-03-26T01:52:55.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:54 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:52:55.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:54 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi069.spgwgo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T01:52:55.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:54 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi069.spgwgo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T01:52:55.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:54 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:52:55.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:54 smithi069 ceph-mon[27071]: Deploying daemon mds.foofs.smithi069.spgwgo on smithi069 2024-03-26T01:52:56.100 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:55 smithi019 ceph-mon[24816]: pgmap v220: 65 pgs: 2 creating+activating, 63 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:56.100 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:55 smithi019 ceph-mon[24816]: mds.? [v2:172.21.15.19:6834/3267072550,v1:172.21.15.19:6835/3267072550] up:active 2024-03-26T01:52:56.100 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:55 smithi019 ceph-mon[24816]: fsmap foofs:1 {0=foofs.smithi019.lmryom=up:active} 2024-03-26T01:52:56.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:55 smithi069 ceph-mon[27071]: pgmap v220: 65 pgs: 2 creating+activating, 63 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:52:56.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:55 smithi069 ceph-mon[27071]: mds.? [v2:172.21.15.19:6834/3267072550,v1:172.21.15.19:6835/3267072550] up:active 2024-03-26T01:52:56.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:55 smithi069 ceph-mon[27071]: fsmap foofs:1 {0=foofs.smithi019.lmryom=up:active} 2024-03-26T01:52:56.637 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-26T01:52:56.638 INFO:teuthology.orchestra.run.smithi019.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-03-26T01:46:13.250596Z", "last_refresh": "2024-03-26T01:49:57.727247Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-03-26T01:46:06.931254Z", "last_refresh": "2024-03-26T01:49:57.727359Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-03-26T01:46:09.898597Z", "last_refresh": "2024-03-26T01:49:57.727430Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-03-26T01:52:48.832653Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-03-26T01:52:48.827688Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-03-26T01:46:05.586775Z", "last_refresh": "2024-03-26T01:49:57.727099Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi019:172.21.15.19=smithi019", "smithi069:172.21.15.69=smithi069"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-03-26T01:46:56.697362Z", "last_refresh": "2024-03-26T01:49:57.726888Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-03-26T01:46:11.134728Z", "last_refresh": "2024-03-26T01:49:57.727496Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-03-26T01:49:57.727624Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-03-26T01:46:08.313914Z", "last_refresh": "2024-03-26T01:49:57.727560Z", "ports": [9095], "running": 1, "size": 1}}] 2024-03-26T01:52:57.268 INFO:tasks.cephadm:mds.foofs has 0/2 2024-03-26T01:52:57.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:57 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:52:57.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:57 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:52:57.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:57 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:52:57.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:57 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:52:57.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:57 smithi019 ceph-mon[24816]: from='client.14484 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T01:52:57.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:57 smithi019 ceph-mon[24816]: pgmap v221: 65 pgs: 65 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 645 B/s wr, 2 op/s 2024-03-26T01:52:57.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:57 smithi019 ceph-mon[24816]: mds.? [v2:172.21.15.69:6832/641242727,v1:172.21.15.69:6833/641242727] up:boot 2024-03-26T01:52:57.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:57 smithi019 ceph-mon[24816]: fsmap foofs:1 {0=foofs.smithi019.lmryom=up:active} 1 up:standby 2024-03-26T01:52:57.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:57 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi069.spgwgo"}]: dispatch 2024-03-26T01:52:57.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:57 smithi019 ceph-mon[24816]: fsmap foofs:1 {0=foofs.smithi019.lmryom=up:active} 1 up:standby 2024-03-26T01:52:57.773 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:57 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:52:57.773 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:57 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:52:57.773 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:57 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:52:57.773 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:57 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:52:57.773 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:57 smithi069 ceph-mon[27071]: from='client.14484 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T01:52:57.773 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:57 smithi069 ceph-mon[27071]: pgmap v221: 65 pgs: 65 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 645 B/s wr, 2 op/s 2024-03-26T01:52:57.774 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:57 smithi069 ceph-mon[27071]: mds.? [v2:172.21.15.69:6832/641242727,v1:172.21.15.69:6833/641242727] up:boot 2024-03-26T01:52:57.774 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:57 smithi069 ceph-mon[27071]: fsmap foofs:1 {0=foofs.smithi019.lmryom=up:active} 1 up:standby 2024-03-26T01:52:57.774 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:57 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi069.spgwgo"}]: dispatch 2024-03-26T01:52:57.774 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:57 smithi069 ceph-mon[27071]: fsmap foofs:1 {0=foofs.smithi019.lmryom=up:active} 1 up:standby 2024-03-26T01:52:58.270 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph orch ls -f json 2024-03-26T01:52:59.764 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:59 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:52:59.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:59 smithi019 ceph-mon[24816]: pgmap v222: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s wr, 4 op/s 2024-03-26T01:52:59.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:52:59 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:52:59.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:59 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:52:59.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:59 smithi069 ceph-mon[27071]: pgmap v222: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s wr, 4 op/s 2024-03-26T01:52:59.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:52:59 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:00.906 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-26T01:53:00.906 INFO:teuthology.orchestra.run.smithi019.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-03-26T01:46:13.250596Z", "last_refresh": "2024-03-26T01:49:57.727247Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-03-26T01:46:06.931254Z", "last_refresh": "2024-03-26T01:49:57.727359Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-03-26T01:46:09.898597Z", "last_refresh": "2024-03-26T01:49:57.727430Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-03-26T01:52:48.832653Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-03-26T01:52:48.827688Z", "last_refresh": "2024-03-26T01:52:58.466658Z", "running": 1, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-03-26T01:46:05.586775Z", "last_refresh": "2024-03-26T01:49:57.727099Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi019:172.21.15.19=smithi019", "smithi069:172.21.15.69=smithi069"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-03-26T01:46:56.697362Z", "last_refresh": "2024-03-26T01:49:57.726888Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-03-26T01:46:11.134728Z", "last_refresh": "2024-03-26T01:49:57.727496Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-03-26T01:49:57.727624Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-03-26T01:46:08.313914Z", "last_refresh": "2024-03-26T01:49:57.727560Z", "ports": [9095], "running": 1, "size": 1}}] 2024-03-26T01:53:01.785 INFO:tasks.cephadm:mds.foofs has 1/2 2024-03-26T01:53:02.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:01 smithi019 ceph-mon[24816]: pgmap v223: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-03-26T01:53:02.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:01 smithi019 ceph-mon[24816]: from='client.14488 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T01:53:02.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:01 smithi069 ceph-mon[27071]: pgmap v223: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-03-26T01:53:02.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:01 smithi069 ceph-mon[27071]: from='client.14488 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T01:53:02.786 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph orch ls -f json 2024-03-26T01:53:04.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:03 smithi069 ceph-mon[27071]: pgmap v224: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-26T01:53:04.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:03 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:04.223 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:03 smithi019 ceph-mon[24816]: pgmap v224: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-26T01:53:04.223 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:03 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:04.978 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-26T01:53:04.978 INFO:teuthology.orchestra.run.smithi019.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-03-26T01:46:13.250596Z", "last_refresh": "2024-03-26T01:53:04.853716Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-03-26T01:46:06.931254Z", "last_refresh": "2024-03-26T01:52:58.466018Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-03-26T01:46:09.898597Z", "last_refresh": "2024-03-26T01:53:04.853871Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-03-26T01:52:48.832653Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-03-26T01:52:48.827688Z", "last_refresh": "2024-03-26T01:52:58.466658Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-03-26T01:46:05.586775Z", "last_refresh": "2024-03-26T01:52:58.466185Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi019:172.21.15.19=smithi019", "smithi069:172.21.15.69=smithi069"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-03-26T01:46:56.697362Z", "last_refresh": "2024-03-26T01:52:58.466262Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-03-26T01:46:11.134728Z", "last_refresh": "2024-03-26T01:52:58.466330Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-03-26T01:52:58.466397Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-03-26T01:46:08.313914Z", "last_refresh": "2024-03-26T01:53:04.854012Z", "ports": [9095], "running": 1, "size": 1}}] 2024-03-26T01:53:05.723 INFO:tasks.cephadm:mds.foofs has 2/2 2024-03-26T01:53:05.724 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-26T01:53:05.733 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi019.front.sepia.ceph.com 2024-03-26T01:53:05.733 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- bash -c 'ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2' 2024-03-26T01:53:06.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:05 smithi019 ceph-mon[24816]: pgmap v225: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-26T01:53:06.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:05 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:06.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:05 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:06.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:05 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:53:06.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:05 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:53:06.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:05 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:53:06.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:05 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:06.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:05 smithi019 ceph-mon[24816]: from='client.14492 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T01:53:06.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:05 smithi069 ceph-mon[27071]: pgmap v225: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-26T01:53:06.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:05 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:06.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:05 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:06.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:05 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:53:06.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:05 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:53:06.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:05 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:53:06.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:05 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:06.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:05 smithi069 ceph-mon[27071]: from='client.14492 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T01:53:08.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:08 smithi069 ceph-mon[27071]: pgmap v226: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-26T01:53:08.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:08 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-03-26T01:53:08.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:08 smithi019 ceph-mon[24816]: pgmap v226: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-26T01:53:08.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:08 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-03-26T01:53:09.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:09 smithi069 ceph-mon[27071]: from='client.14496 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:53:09.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:09 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-03-26T01:53:09.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:09 smithi069 ceph-mon[27071]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T01:53:09.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:09 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-03-26T01:53:09.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:09 smithi019 ceph-mon[24816]: from='client.14496 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:53:09.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:09 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-03-26T01:53:09.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:09 smithi019 ceph-mon[24816]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T01:53:09.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:09 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-03-26T01:53:09.804 INFO:teuthology.orchestra.run.smithi019.stdout:NFS Cluster Created Successfully 2024-03-26T01:53:10.182 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:10 smithi019 ceph-mon[24816]: pgmap v227: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 682 B/s wr, 1 op/s 2024-03-26T01:53:10.182 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:10 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-03-26T01:53:10.182 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:10 smithi019 ceph-mon[24816]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T01:53:10.182 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:10 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:10.182 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:10 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:53:10.182 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:10 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:53:10.182 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:10 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:53:10.182 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:10 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:10.182 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:10 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:10.182 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:10 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi019.qbizxo", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-26T01:53:10.182 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:10 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi019.qbizxo", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-03-26T01:53:10.183 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:10 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-26T01:53:10.183 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:10 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-26T01:53:10.183 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:10 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:53:10.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:10 smithi069 ceph-mon[27071]: pgmap v227: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 682 B/s wr, 1 op/s 2024-03-26T01:53:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:10 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-03-26T01:53:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:10 smithi069 ceph-mon[27071]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T01:53:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:10 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:10 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:53:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:10 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:53:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:10 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:53:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:10 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:10 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:10 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi019.qbizxo", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-26T01:53:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:10 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi019.qbizxo", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-03-26T01:53:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:10 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-26T01:53:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:10 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-26T01:53:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:10 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:53:10.627 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- bash -c 'ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake' 2024-03-26T01:53:11.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:11 smithi019 ceph-mon[24816]: Saving service nfs.foo spec with placement count:2 2024-03-26T01:53:11.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:11 smithi019 ceph-mon[24816]: Creating key for client.nfs.foo.0.0.smithi019.qbizxo 2024-03-26T01:53:11.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:11 smithi019 ceph-mon[24816]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-03-26T01:53:11.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:11 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-26T01:53:11.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:11 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-26T01:53:11.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:11 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi019.qbizxo-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T01:53:11.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:11 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi019.qbizxo-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T01:53:11.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:11 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:53:11.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:11 smithi019 ceph-mon[24816]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T01:53:11.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:11 smithi019 ceph-mon[24816]: mgrmap e20: smithi019.ixuvdm(active, since 5m), standbys: smithi069.wpbewu 2024-03-26T01:53:11.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:11 smithi069 ceph-mon[27071]: Saving service nfs.foo spec with placement count:2 2024-03-26T01:53:11.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:11 smithi069 ceph-mon[27071]: Creating key for client.nfs.foo.0.0.smithi019.qbizxo 2024-03-26T01:53:11.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:11 smithi069 ceph-mon[27071]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-03-26T01:53:11.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:11 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-26T01:53:11.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:11 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-26T01:53:11.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:11 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi019.qbizxo-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T01:53:11.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:11 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi019.qbizxo-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T01:53:11.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:11 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:53:11.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:11 smithi069 ceph-mon[27071]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T01:53:11.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:11 smithi069 ceph-mon[27071]: mgrmap e20: smithi019.ixuvdm(active, since 5m), standbys: smithi069.wpbewu 2024-03-26T01:53:12.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:12 smithi069 ceph-mon[27071]: Rados config object exists: conf-nfs.foo 2024-03-26T01:53:12.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:12 smithi069 ceph-mon[27071]: Creating key for client.nfs.foo.0.0.smithi019.qbizxo-rgw 2024-03-26T01:53:12.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:12 smithi069 ceph-mon[27071]: Deploying daemon nfs.foo.0.0.smithi019.qbizxo on smithi019 2024-03-26T01:53:12.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:12 smithi069 ceph-mon[27071]: pgmap v230: 97 pgs: 27 unknown, 70 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:53:12.453 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:12 smithi019 ceph-mon[24816]: Rados config object exists: conf-nfs.foo 2024-03-26T01:53:12.453 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:12 smithi019 ceph-mon[24816]: Creating key for client.nfs.foo.0.0.smithi019.qbizxo-rgw 2024-03-26T01:53:12.453 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:12 smithi019 ceph-mon[24816]: Deploying daemon nfs.foo.0.0.smithi019.qbizxo on smithi019 2024-03-26T01:53:12.454 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:12 smithi019 ceph-mon[24816]: pgmap v230: 97 pgs: 27 unknown, 70 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:53:13.737 INFO:teuthology.orchestra.run.smithi019.stderr:Invalid command: Unexpected argument '--clusterid' 2024-03-26T01:53:13.737 INFO:teuthology.orchestra.run.smithi019.stderr:nfs export create cephfs [] [--readonly] [--client_addr ...] [--squash ] : Create a CephFS export 2024-03-26T01:53:13.737 INFO:teuthology.orchestra.run.smithi019.stderr:Error EINVAL: invalid command 2024-03-26T01:53:14.098 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T01:53:14.098 INFO:teuthology.orchestra.run.smithi019.stdout: "bind": "/fake", 2024-03-26T01:53:14.098 INFO:teuthology.orchestra.run.smithi019.stdout: "fs": "foofs", 2024-03-26T01:53:14.098 INFO:teuthology.orchestra.run.smithi019.stdout: "path": "/", 2024-03-26T01:53:14.098 INFO:teuthology.orchestra.run.smithi019.stdout: "cluster": "foo", 2024-03-26T01:53:14.098 INFO:teuthology.orchestra.run.smithi019.stdout: "mode": "RW" 2024-03-26T01:53:14.098 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T01:53:14.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:14 smithi019 ceph-mon[24816]: pgmap v232: 97 pgs: 18 unknown, 79 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:53:14.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:14 smithi069 ceph-mon[27071]: pgmap v232: 97 pgs: 18 unknown, 79 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-03-26T01:53:14.919 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- bash -c 'while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done' 2024-03-26T01:53:15.233 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:15 smithi019 ceph-mon[24816]: from='client.14516 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:53:15.233 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:15 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-03-26T01:53:15.233 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:15 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-03-26T01:53:15.233 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:15 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:15.233 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:15 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi069.uawdhc", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-26T01:53:15.234 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:15 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi069.uawdhc", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-03-26T01:53:15.234 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:15 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-26T01:53:15.234 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:15 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-26T01:53:15.234 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:15 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:53:15.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:15 smithi069 ceph-mon[27071]: from='client.14516 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:53:15.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:15 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-03-26T01:53:15.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:15 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-03-26T01:53:15.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:15 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:15.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:15 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi069.uawdhc", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-26T01:53:15.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:15 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi069.uawdhc", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-03-26T01:53:15.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:15 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-26T01:53:15.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:15 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-26T01:53:15.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:15 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:53:16.356 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:16 smithi019 ceph-mon[24816]: Creating key for client.nfs.foo.1.0.smithi069.uawdhc 2024-03-26T01:53:16.356 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:16 smithi019 ceph-mon[24816]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-03-26T01:53:16.356 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:16 smithi019 ceph-mon[24816]: pgmap v233: 97 pgs: 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-03-26T01:53:16.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:16 smithi069 ceph-mon[27071]: Creating key for client.nfs.foo.1.0.smithi069.uawdhc 2024-03-26T01:53:16.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:16 smithi069 ceph-mon[27071]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-03-26T01:53:16.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:16 smithi069 ceph-mon[27071]: pgmap v233: 97 pgs: 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-03-26T01:53:18.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:18 smithi069 ceph-mon[27071]: pgmap v234: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 128 B/s rd, 384 B/s wr, 0 op/s 2024-03-26T01:53:18.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:18 smithi069 ceph-mon[27071]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:53:18.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:18 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-26T01:53:18.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:18 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-26T01:53:18.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:18 smithi069 ceph-mon[27071]: Rados config object exists: conf-nfs.foo 2024-03-26T01:53:18.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:18 smithi069 ceph-mon[27071]: Creating key for client.nfs.foo.1.0.smithi069.uawdhc-rgw 2024-03-26T01:53:18.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:18 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi069.uawdhc-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T01:53:18.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:18 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi069.uawdhc-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T01:53:18.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:18 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:53:18.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:18 smithi069 ceph-mon[27071]: Deploying daemon nfs.foo.1.0.smithi069.uawdhc on smithi069 2024-03-26T01:53:18.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:18 smithi019 ceph-mon[24816]: pgmap v234: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 128 B/s rd, 384 B/s wr, 0 op/s 2024-03-26T01:53:18.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:18 smithi019 ceph-mon[24816]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:53:18.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:18 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-26T01:53:18.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:18 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-26T01:53:18.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:18 smithi019 ceph-mon[24816]: Rados config object exists: conf-nfs.foo 2024-03-26T01:53:18.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:18 smithi019 ceph-mon[24816]: Creating key for client.nfs.foo.1.0.smithi069.uawdhc-rgw 2024-03-26T01:53:18.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:18 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi069.uawdhc-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T01:53:18.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:18 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi069.uawdhc-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T01:53:18.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:18 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:53:18.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:18 smithi019 ceph-mon[24816]: Deploying daemon nfs.foo.1.0.smithi069.uawdhc on smithi069 2024-03-26T01:53:19.254 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:19 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]: dispatch 2024-03-26T01:53:19.254 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:19 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-03-26T01:53:19.254 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:19 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-03-26T01:53:19.255 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:19 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-03-26T01:53:19.255 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:19 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/trash_purge_schedule"}]: dispatch 2024-03-26T01:53:19.255 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:19 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:53:19.514 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:19 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]: dispatch 2024-03-26T01:53:19.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:19 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-03-26T01:53:19.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:19 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-03-26T01:53:19.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:19 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-03-26T01:53:19.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:19 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/trash_purge_schedule"}]: dispatch 2024-03-26T01:53:19.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:19 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:53:20.317 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:20 smithi019 ceph-mon[24816]: from='client.14546 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:53:20.317 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:20 smithi019 ceph-mon[24816]: pgmap v235: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T01:53:20.317 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:20 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]': finished 2024-03-26T01:53:20.317 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:20 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-03-26T01:53:20.317 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:20 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-03-26T01:53:20.317 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:20 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-03-26T01:53:20.317 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:20 smithi019 ceph-mon[24816]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T01:53:20.318 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:20 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:20.318 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:20 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:53:20.318 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:20 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:53:20.318 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:20 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:53:20.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:20 smithi069 ceph-mon[27071]: from='client.14546 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:53:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:20 smithi069 ceph-mon[27071]: pgmap v235: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T01:53:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:20 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]': finished 2024-03-26T01:53:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:20 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-03-26T01:53:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:20 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-03-26T01:53:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:20 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-03-26T01:53:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:20 smithi069 ceph-mon[27071]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T01:53:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:20 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:20 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:53:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:20 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:53:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:20 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:53:21.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:21 smithi069 ceph-mon[27071]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:53:21.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:21 smithi069 ceph-mon[27071]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T01:53:21.514 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:21 smithi019 ceph-mon[24816]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:53:21.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:21 smithi019 ceph-mon[24816]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T01:53:22.257 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:22 smithi069 ceph-mon[27071]: pgmap v238: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-03-26T01:53:22.257 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:22 smithi069 ceph-mon[27071]: from='client.14566 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:53:22.353 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:22 smithi019 ceph-mon[24816]: pgmap v238: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-03-26T01:53:22.353 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:22 smithi019 ceph-mon[24816]: from='client.14566 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:53:23.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:23 smithi069 ceph-mon[27071]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:53:23.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:23 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:23.514 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:23 smithi019 ceph-mon[24816]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:53:23.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:23 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:24.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:24 smithi019 ceph-mon[24816]: pgmap v239: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 2.1 KiB/s wr, 4 op/s 2024-03-26T01:53:24.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:24 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:24.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:24 smithi069 ceph-mon[27071]: pgmap v239: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 2.1 KiB/s wr, 4 op/s 2024-03-26T01:53:24.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:24 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:24.832 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo ?:2049 2/2 2s ago 15s count:2 2024-03-26T01:53:25.264 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:25 smithi019 ceph-mon[24816]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:53:25.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:25 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:25.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:25 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:25.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:25 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:25.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:25 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:53:25.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:25 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:53:25.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:25 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:53:25.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:25 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:25.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:25 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:25.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:25 smithi069 ceph-mon[27071]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:53:25.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:25 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:25.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:25 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:25.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:25 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:25.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:25 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:53:25.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:25 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:53:25.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:25 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:53:25.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:25 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:25.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:25 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:25.778 INFO:teuthology.run_tasks:Running task vip.exec... 2024-03-26T01:53:25.787 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi019.front.sepia.ceph.com 2024-03-26T01:53:25.787 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mkdir /mnt/foo' 2024-03-26T01:53:25.816 INFO:teuthology.orchestra.run.smithi019.stderr:+ mkdir /mnt/foo 2024-03-26T01:53:25.835 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done' 2024-03-26T01:53:25.902 INFO:teuthology.orchestra.run.smithi019.stderr:++ hostname 2024-03-26T01:53:25.903 INFO:teuthology.orchestra.run.smithi019.stderr:+ mount -t nfs smithi019:/fake /mnt/foo -o sync 2024-03-26T01:53:26.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:26 smithi069 ceph-mon[27071]: pgmap v240: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.0 KiB/s wr, 8 op/s; 2 B/s, 0 objects/s recovering 2024-03-26T01:53:26.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:26 smithi069 ceph-mon[27071]: from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:53:26.373 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:26 smithi019 ceph-mon[24816]: pgmap v240: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.0 KiB/s wr, 8 op/s; 2 B/s, 0 objects/s recovering 2024-03-26T01:53:26.373 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:26 smithi019 ceph-mon[24816]: from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:53:26.391 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'echo test > /mnt/foo/testfile' 2024-03-26T01:53:26.458 INFO:teuthology.orchestra.run.smithi019.stderr:+ echo test 2024-03-26T01:53:26.467 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c sync 2024-03-26T01:53:26.534 INFO:teuthology.orchestra.run.smithi019.stderr:+ sync 2024-03-26T01:53:26.879 INFO:teuthology.run_tasks:Running task parallel... 2024-03-26T01:53:26.891 INFO:teuthology.task.parallel:starting parallel... 2024-03-26T01:53:26.891 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-26T01:53:26.906 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-26T01:53:26.907 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi019.front.sepia.ceph.com 2024-03-26T01:53:26.907 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-03-26T01:53:26.908 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-26T01:53:26.908 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-03-26T01:53:26.909 INFO:teuthology.task.exec:Executing custom commands... 2024-03-26T01:53:26.909 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi019.front.sepia.ceph.com 2024-03-26T01:53:26.909 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 600 || true' 2024-03-26T01:53:26.959 INFO:teuthology.orchestra.run.smithi019.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-03-26T01:53:26.959 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-26T01:53:26.959 INFO:teuthology.orchestra.run.smithi019.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-03-26T01:53:26.960 INFO:teuthology.orchestra.run.smithi019.stdout:3 of 5 processes prepared for launch 0 sec 2024-03-26T01:53:27.063 INFO:teuthology.orchestra.run.smithi019.stdout:5 of 5 processes prepared for launch 0 sec 2024-03-26T01:53:27.063 INFO:teuthology.orchestra.run.smithi019.stdout:releasing clients 2024-03-26T01:53:28.060 INFO:teuthology.orchestra.run.smithi019.stdout: 5 234 44.88 MB/sec warmup 1 sec latency 29.793 ms 2024-03-26T01:53:28.253 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:28 smithi019 ceph-mon[24816]: pgmap v241: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 5 op/s; 2 B/s, 0 objects/s recovering 2024-03-26T01:53:28.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:28 smithi069 ceph-mon[27071]: pgmap v241: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 5 op/s; 2 B/s, 0 objects/s recovering 2024-03-26T01:53:29.060 INFO:teuthology.orchestra.run.smithi019.stdout: 5 381 33.85 MB/sec warmup 2 sec latency 31.021 ms 2024-03-26T01:53:30.060 INFO:teuthology.orchestra.run.smithi019.stdout: 5 511 29.52 MB/sec warmup 3 sec latency 45.416 ms 2024-03-26T01:53:30.344 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:30 smithi019 ceph-mon[24816]: pgmap v242: 97 pgs: 97 active+clean; 23 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 2.3 MiB/s wr, 46 op/s; 2 B/s, 0 objects/s recovering 2024-03-26T01:53:30.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:30 smithi069 ceph-mon[27071]: pgmap v242: 97 pgs: 97 active+clean; 23 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 2.3 MiB/s wr, 46 op/s; 2 B/s, 0 objects/s recovering 2024-03-26T01:53:30.757 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-03-26T01:53:31.060 INFO:teuthology.orchestra.run.smithi019.stdout: 5 751 26.65 MB/sec warmup 4 sec latency 45.745 ms 2024-03-26T01:53:32.060 INFO:teuthology.orchestra.run.smithi019.stdout: 5 1167 23.02 MB/sec warmup 5 sec latency 63.096 ms 2024-03-26T01:53:32.112 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:32 smithi019 ceph-mon[24816]: pgmap v243: 97 pgs: 97 active+clean; 32 MiB data, 66 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2.9 MiB/s wr, 58 op/s; 2 B/s, 0 objects/s recovering 2024-03-26T01:53:32.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:32 smithi069 ceph-mon[27071]: pgmap v243: 97 pgs: 97 active+clean; 32 MiB data, 66 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2.9 MiB/s wr, 58 op/s; 2 B/s, 0 objects/s recovering 2024-03-26T01:53:33.060 INFO:teuthology.orchestra.run.smithi019.stdout: 5 2272 22.77 MB/sec warmup 6 sec latency 18.837 ms 2024-03-26T01:53:34.061 INFO:teuthology.orchestra.run.smithi019.stdout: 5 3050 24.13 MB/sec warmup 7 sec latency 19.465 ms 2024-03-26T01:53:34.121 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:34 smithi019 ceph-mon[24816]: pgmap v244: 97 pgs: 97 active+clean; 80 MiB data, 109 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 6.9 MiB/s wr, 259 op/s; 1 B/s, 0 objects/s recovering 2024-03-26T01:53:34.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:34 smithi069 ceph-mon[27071]: pgmap v244: 97 pgs: 97 active+clean; 80 MiB data, 109 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 6.9 MiB/s wr, 259 op/s; 1 B/s, 0 objects/s recovering 2024-03-26T01:53:35.060 INFO:teuthology.orchestra.run.smithi019.stdout: 5 3736 24.54 MB/sec warmup 8 sec latency 17.707 ms 2024-03-26T01:53:35.641 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'ceph config set global log_to_journald false --force' 2024-03-26T01:53:36.060 INFO:teuthology.orchestra.run.smithi019.stdout: 5 4304 22.45 MB/sec warmup 9 sec latency 24.027 ms 2024-03-26T01:53:36.129 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:36 smithi019 ceph-mon[24816]: pgmap v245: 97 pgs: 97 active+clean; 111 MiB data, 186 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 9.9 MiB/s wr, 450 op/s; 1 B/s, 0 objects/s recovering 2024-03-26T01:53:36.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:36 smithi069 ceph-mon[27071]: pgmap v245: 97 pgs: 97 active+clean; 111 MiB data, 186 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 9.9 MiB/s wr, 450 op/s; 1 B/s, 0 objects/s recovering 2024-03-26T01:53:37.061 INFO:teuthology.orchestra.run.smithi019.stdout: 5 4817 21.38 MB/sec warmup 10 sec latency 29.475 ms 2024-03-26T01:53:38.061 INFO:teuthology.orchestra.run.smithi019.stdout: 5 5667 20.80 MB/sec warmup 11 sec latency 50.231 ms 2024-03-26T01:53:38.157 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:38 smithi019 ceph-mon[24816]: pgmap v246: 97 pgs: 97 active+clean; 121 MiB data, 217 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 11 MiB/s wr, 619 op/s 2024-03-26T01:53:38.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:38 smithi069 ceph-mon[27071]: pgmap v246: 97 pgs: 97 active+clean; 121 MiB data, 217 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 11 MiB/s wr, 619 op/s 2024-03-26T01:53:39.061 INFO:teuthology.orchestra.run.smithi019.stdout: 5 6321 21.11 MB/sec warmup 12 sec latency 63.413 ms 2024-03-26T01:53:40.061 INFO:teuthology.orchestra.run.smithi019.stdout: 5 7300 22.52 MB/sec warmup 13 sec latency 12.592 ms 2024-03-26T01:53:40.170 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:40 smithi019 ceph-mon[24816]: pgmap v247: 97 pgs: 97 active+clean; 132 MiB data, 234 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 14 MiB/s wr, 808 op/s 2024-03-26T01:53:40.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:40 smithi069 ceph-mon[27071]: pgmap v247: 97 pgs: 97 active+clean; 132 MiB data, 234 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 14 MiB/s wr, 808 op/s 2024-03-26T01:53:40.863 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'ceph mgr module enable nfs --force' 2024-03-26T01:53:41.061 INFO:teuthology.orchestra.run.smithi019.stdout: 5 7913 21.31 MB/sec warmup 14 sec latency 29.577 ms 2024-03-26T01:53:42.061 INFO:teuthology.orchestra.run.smithi019.stdout: 5 8210 20.39 MB/sec warmup 15 sec latency 80.599 ms 2024-03-26T01:53:42.227 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:42 smithi019 ceph-mon[24816]: pgmap v248: 97 pgs: 97 active+clean; 133 MiB data, 238 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 12 MiB/s wr, 789 op/s 2024-03-26T01:53:42.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:42 smithi069 ceph-mon[27071]: pgmap v248: 97 pgs: 97 active+clean; 133 MiB data, 238 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 12 MiB/s wr, 789 op/s 2024-03-26T01:53:43.061 INFO:teuthology.orchestra.run.smithi019.stdout: 5 8733 20.13 MB/sec warmup 16 sec latency 105.180 ms 2024-03-26T01:53:44.061 INFO:teuthology.orchestra.run.smithi019.stdout: 5 9364 19.20 MB/sec warmup 17 sec latency 27.452 ms 2024-03-26T01:53:44.237 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:44 smithi019 ceph-mon[24816]: pgmap v249: 97 pgs: 97 active+clean; 138 MiB data, 312 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s rd, 14 MiB/s wr, 981 op/s 2024-03-26T01:53:44.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:44 smithi069 ceph-mon[27071]: pgmap v249: 97 pgs: 97 active+clean; 138 MiB data, 312 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s rd, 14 MiB/s wr, 981 op/s 2024-03-26T01:53:45.061 INFO:teuthology.orchestra.run.smithi019.stdout: 5 9655 18.77 MB/sec warmup 18 sec latency 47.255 ms 2024-03-26T01:53:45.144 INFO:teuthology.orchestra.run.smithi019.stderr:module 'nfs' is already enabled 2024-03-26T01:53:45.252 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:45 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/1017980114' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-03-26T01:53:45.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:45 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/1017980114' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-03-26T01:53:46.061 INFO:teuthology.orchestra.run.smithi019.stdout: 5 10110 18.96 MB/sec warmup 19 sec latency 77.327 ms 2024-03-26T01:53:46.282 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:46 smithi019 ceph-mon[24816]: pgmap v250: 97 pgs: 97 active+clean; 143 MiB data, 354 MiB used, 715 GiB / 715 GiB avail; 8.2 KiB/s rd, 11 MiB/s wr, 919 op/s 2024-03-26T01:53:46.282 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:46 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/1017980114' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-03-26T01:53:46.282 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:46 smithi019 ceph-mon[24816]: mgrmap e21: smithi019.ixuvdm(active, since 6m), standbys: smithi069.wpbewu 2024-03-26T01:53:46.326 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-03-26T01:53:46.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:46 smithi069 ceph-mon[27071]: pgmap v250: 97 pgs: 97 active+clean; 143 MiB data, 354 MiB used, 715 GiB / 715 GiB avail; 8.2 KiB/s rd, 11 MiB/s wr, 919 op/s 2024-03-26T01:53:46.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:46 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/1017980114' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-03-26T01:53:46.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:46 smithi069 ceph-mon[27071]: mgrmap e21: smithi019.ixuvdm(active, since 6m), standbys: smithi069.wpbewu 2024-03-26T01:53:47.061 INFO:teuthology.orchestra.run.smithi019.stdout: 5 10522 18.58 MB/sec warmup 20 sec latency 81.374 ms 2024-03-26T01:53:48.061 INFO:teuthology.orchestra.run.smithi019.stdout: 5 10950 18.59 MB/sec warmup 21 sec latency 40.820 ms 2024-03-26T01:53:48.341 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:48 smithi019 ceph-mon[24816]: pgmap v251: 97 pgs: 97 active+clean; 148 MiB data, 372 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s rd, 9.4 MiB/s wr, 906 op/s 2024-03-26T01:53:48.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:48 smithi069 ceph-mon[27071]: pgmap v251: 97 pgs: 97 active+clean; 148 MiB data, 372 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s rd, 9.4 MiB/s wr, 906 op/s 2024-03-26T01:53:49.061 INFO:teuthology.orchestra.run.smithi019.stdout: 5 11474 17.96 MB/sec warmup 22 sec latency 32.483 ms 2024-03-26T01:53:49.441 INFO:teuthology.orchestra.run.smithi019.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T01:53:50.061 INFO:teuthology.orchestra.run.smithi019.stdout: 5 12317 18.17 MB/sec warmup 23 sec latency 19.622 ms 2024-03-26T01:53:50.351 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:50 smithi019 ceph-mon[24816]: pgmap v252: 97 pgs: 97 active+clean; 148 MiB data, 417 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 9.4 MiB/s wr, 803 op/s 2024-03-26T01:53:50.351 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:50 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:50.351 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:50 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:53:50.351 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:50 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:53:50.351 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:50 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:53:50.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:50 smithi069 ceph-mon[27071]: pgmap v252: 97 pgs: 97 active+clean; 148 MiB data, 417 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 9.4 MiB/s wr, 803 op/s 2024-03-26T01:53:50.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:50 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:50.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:50 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:53:50.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:50 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:53:50.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:50 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:53:50.622 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-26T01:53:50.623 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi019.front.sepia.ceph.com 2024-03-26T01:53:50.623 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-03-26T01:53:51.061 INFO:teuthology.orchestra.run.smithi019.stdout: 5 13116 17.94 MB/sec warmup 24 sec latency 25.895 ms 2024-03-26T01:53:51.355 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:51 smithi019 ceph-mon[24816]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:53:51.355 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:51 smithi019 ceph-mon[24816]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T01:53:51.355 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:51 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:51.355 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:51 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:51.544 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:51 smithi069 ceph-mon[27071]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:53:51.544 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:51 smithi069 ceph-mon[27071]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T01:53:51.544 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:51 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:51.544 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:51 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:53:52.061 INFO:teuthology.orchestra.run.smithi019.stdout: 5 14011 18.65 MB/sec warmup 25 sec latency 15.327 ms 2024-03-26T01:53:52.360 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:52 smithi019 ceph-mon[24816]: pgmap v253: 97 pgs: 97 active+clean; 149 MiB data, 419 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 6.9 MiB/s wr, 621 op/s 2024-03-26T01:53:52.360 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:52 smithi019 ceph-mon[24816]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T01:53:52.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:52 smithi069 ceph-mon[27071]: pgmap v253: 97 pgs: 97 active+clean; 149 MiB data, 419 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 6.9 MiB/s wr, 621 op/s 2024-03-26T01:53:52.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:52 smithi069 ceph-mon[27071]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T01:53:53.062 INFO:teuthology.orchestra.run.smithi019.stdout: 5 14626 18.74 MB/sec warmup 26 sec latency 15.960 ms 2024-03-26T01:53:54.062 INFO:teuthology.orchestra.run.smithi019.stdout: 5 15241 18.38 MB/sec warmup 27 sec latency 23.242 ms 2024-03-26T01:53:54.262 INFO:teuthology.orchestra.run.smithi019.stdout:true 2024-03-26T01:53:54.377 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:54 smithi019 ceph-mon[24816]: pgmap v254: 97 pgs: 97 active+clean; 154 MiB data, 457 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 8.7 MiB/s wr, 805 op/s 2024-03-26T01:53:54.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:54 smithi069 ceph-mon[27071]: pgmap v254: 97 pgs: 97 active+clean; 154 MiB data, 457 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 8.7 MiB/s wr, 805 op/s 2024-03-26T01:53:55.062 INFO:teuthology.orchestra.run.smithi019.stdout: 5 15890 18.34 MB/sec warmup 28 sec latency 25.009 ms 2024-03-26T01:53:55.299 INFO:teuthology.orchestra.run.smithi019.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T01:53:55.299 INFO:teuthology.orchestra.run.smithi019.stdout:alertmanager.smithi019 smithi019 *:9093,9094 running (5m) 30s ago 7m 23.2M - ba2b418f427c b72e14ab8874 2024-03-26T01:53:55.299 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi019 smithi019 running (7m) 30s ago 7m 6405k - 17.2.0 e1d6a67b021e 713db693d5c1 2024-03-26T01:53:55.299 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi069 smithi069 running (6m) 33s ago 6m 6405k - 17.2.0 e1d6a67b021e a87d27e2b74b 2024-03-26T01:53:55.299 INFO:teuthology.orchestra.run.smithi019.stdout:grafana.smithi019 smithi019 *:3000 running (5m) 30s ago 6m 52.5M - 8.3.5 dad864ee21e9 81e9f452f143 2024-03-26T01:53:55.299 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi019.lmryom smithi019 running (62s) 30s ago 61s 13.2M - 17.2.0 e1d6a67b021e 91a7c2942715 2024-03-26T01:53:55.300 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi069.spgwgo smithi069 running (59s) 33s ago 58s 11.9M - 17.2.0 e1d6a67b021e 94d96338dbe1 2024-03-26T01:53:55.300 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi019.ixuvdm smithi019 *:9283 running (8m) 30s ago 8m 460M - 17.2.0 e1d6a67b021e 1d13a7db0753 2024-03-26T01:53:55.300 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi069.wpbewu smithi069 *:8443,9283 running (6m) 33s ago 6m 412M - 17.2.0 e1d6a67b021e 0a4bd07111bb 2024-03-26T01:53:55.300 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi019 smithi019 running (8m) 30s ago 8m 46.3M 2048M 17.2.0 e1d6a67b021e b6908b497829 2024-03-26T01:53:55.300 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi069 smithi069 running (6m) 33s ago 6m 39.2M 2048M 17.2.0 e1d6a67b021e 2d51b7b7bb32 2024-03-26T01:53:55.300 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.0.0.smithi019.qbizxo smithi019 *:2049 running (41s) 30s ago 41s 60.0M - 4.0 e1d6a67b021e cd6628a9075d 2024-03-26T01:53:55.300 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.1.0.smithi069.uawdhc smithi069 *:2049 running (36s) 33s ago 35s 14.2M - 4.0 e1d6a67b021e 6ee3695e7a10 2024-03-26T01:53:55.300 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi019 smithi019 *:9100 running (6m) 30s ago 6m 17.0M - 1dbe0e931976 8094999d4d38 2024-03-26T01:53:55.300 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi069 smithi069 *:9100 running (6m) 33s ago 6m 17.7M - 1dbe0e931976 46e719c0de82 2024-03-26T01:53:55.300 INFO:teuthology.orchestra.run.smithi019.stdout:osd.0 smithi019 running (5m) 30s ago 5m 43.3M 3217M 17.2.0 e1d6a67b021e c1a797230ab6 2024-03-26T01:53:55.300 INFO:teuthology.orchestra.run.smithi019.stdout:osd.1 smithi019 running (4m) 30s ago 4m 45.5M 3217M 17.2.0 e1d6a67b021e 31d3489c7188 2024-03-26T01:53:55.300 INFO:teuthology.orchestra.run.smithi019.stdout:osd.2 smithi019 running (4m) 30s ago 4m 39.9M 3217M 17.2.0 e1d6a67b021e f549115e7685 2024-03-26T01:53:55.300 INFO:teuthology.orchestra.run.smithi019.stdout:osd.3 smithi019 running (4m) 30s ago 4m 41.6M 3217M 17.2.0 e1d6a67b021e 154af68d03e1 2024-03-26T01:53:55.301 INFO:teuthology.orchestra.run.smithi019.stdout:osd.4 smithi069 running (3m) 33s ago 3m 40.4M 3984M 17.2.0 e1d6a67b021e 4fd5e853f8bf 2024-03-26T01:53:55.301 INFO:teuthology.orchestra.run.smithi019.stdout:osd.5 smithi069 running (3m) 33s ago 3m 38.9M 3984M 17.2.0 e1d6a67b021e 932e401d87c9 2024-03-26T01:53:55.301 INFO:teuthology.orchestra.run.smithi019.stdout:osd.6 smithi069 running (2m) 33s ago 2m 39.3M 3984M 17.2.0 e1d6a67b021e b123f75154c8 2024-03-26T01:53:55.301 INFO:teuthology.orchestra.run.smithi019.stdout:osd.7 smithi069 running (2m) 33s ago 2m 43.6M 3984M 17.2.0 e1d6a67b021e d175fbc9ff8a 2024-03-26T01:53:55.301 INFO:teuthology.orchestra.run.smithi019.stdout:prometheus.smithi019 smithi019 *:9095 running (5m) 30s ago 5m 62.8M - 514e6a882f6e 33689b2df9a5 2024-03-26T01:53:55.916 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T01:53:55.917 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": { 2024-03-26T01:53:55.917 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T01:53:55.917 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:53:55.917 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": { 2024-03-26T01:53:55.917 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T01:53:55.917 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:53:55.917 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": { 2024-03-26T01:53:55.917 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T01:53:55.917 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:53:55.917 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": { 2024-03-26T01:53:55.917 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T01:53:55.917 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:53:55.917 INFO:teuthology.orchestra.run.smithi019.stdout: "overall": { 2024-03-26T01:53:55.918 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-03-26T01:53:55.918 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-26T01:53:55.918 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T01:53:56.062 INFO:teuthology.orchestra.run.smithi019.stdout: 5 16604 18.11 MB/sec warmup 29 sec latency 24.153 ms 2024-03-26T01:53:56.392 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:56 smithi019 ceph-mon[24816]: from='client.14602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:53:56.392 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:56 smithi019 ceph-mon[24816]: pgmap v255: 97 pgs: 97 active+clean; 159 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 8.6 MiB/s wr, 757 op/s 2024-03-26T01:53:56.392 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:56 smithi019 ceph-mon[24816]: from='client.14606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:53:56.392 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:56 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/4215170949' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T01:53:56.478 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T01:53:56.478 INFO:teuthology.orchestra.run.smithi019.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a", 2024-03-26T01:53:56.479 INFO:teuthology.orchestra.run.smithi019.stdout: "in_progress": true, 2024-03-26T01:53:56.479 INFO:teuthology.orchestra.run.smithi019.stdout: "services_complete": [], 2024-03-26T01:53:56.479 INFO:teuthology.orchestra.run.smithi019.stdout: "progress": "", 2024-03-26T01:53:56.479 INFO:teuthology.orchestra.run.smithi019.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a image" 2024-03-26T01:53:56.479 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T01:53:56.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:56 smithi069 ceph-mon[27071]: from='client.14602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:53:56.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:56 smithi069 ceph-mon[27071]: pgmap v255: 97 pgs: 97 active+clean; 159 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 8.6 MiB/s wr, 757 op/s 2024-03-26T01:53:56.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:56 smithi069 ceph-mon[27071]: from='client.14606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:53:56.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:56 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/4215170949' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T01:53:57.062 INFO:teuthology.orchestra.run.smithi019.stdout: 5 17100 18.24 MB/sec warmup 30 sec latency 28.866 ms 2024-03-26T01:53:57.074 INFO:teuthology.orchestra.run.smithi019.stdout:HEALTH_OK 2024-03-26T01:53:57.424 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:57 smithi019 ceph-mon[24816]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:53:57.424 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:57 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/1282233826' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T01:53:57.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:57 smithi069 ceph-mon[27071]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:53:57.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:57 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/1282233826' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T01:53:58.062 INFO:teuthology.orchestra.run.smithi019.stdout: 5 17658 18.28 MB/sec warmup 31 sec latency 31.317 ms 2024-03-26T01:53:58.184 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:58 smithi019 ceph-mon[24816]: from='client.14618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:53:58.184 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:53:58 smithi019 ceph-mon[24816]: pgmap v256: 97 pgs: 97 active+clean; 162 MiB data, 504 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 7.9 MiB/s wr, 729 op/s 2024-03-26T01:53:58.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:58 smithi069 ceph-mon[27071]: from='client.14618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:53:58.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:53:58 smithi069 ceph-mon[27071]: pgmap v256: 97 pgs: 97 active+clean; 162 MiB data, 504 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 7.9 MiB/s wr, 729 op/s 2024-03-26T01:53:59.062 INFO:teuthology.orchestra.run.smithi019.stdout: 5 18110 18.19 MB/sec warmup 32 sec latency 55.049 ms 2024-03-26T01:54:00.062 INFO:teuthology.orchestra.run.smithi019.stdout: 5 18544 17.75 MB/sec warmup 33 sec latency 25.189 ms 2024-03-26T01:54:00.197 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:00 smithi019 ceph-mon[24816]: pgmap v257: 97 pgs: 97 active+clean; 167 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 9.3 MiB/s wr, 807 op/s 2024-03-26T01:54:00.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:00 smithi069 ceph-mon[27071]: pgmap v257: 97 pgs: 97 active+clean; 167 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 9.3 MiB/s wr, 807 op/s 2024-03-26T01:54:01.062 INFO:teuthology.orchestra.run.smithi019.stdout: 5 19037 17.54 MB/sec warmup 34 sec latency 52.746 ms 2024-03-26T01:54:02.063 INFO:teuthology.orchestra.run.smithi019.stdout: 5 19626 17.46 MB/sec warmup 35 sec latency 25.479 ms 2024-03-26T01:54:02.208 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:02 smithi019 ceph-mon[24816]: pgmap v258: 97 pgs: 97 active+clean; 170 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 8.2 MiB/s wr, 814 op/s 2024-03-26T01:54:02.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:02 smithi069 ceph-mon[27071]: pgmap v258: 97 pgs: 97 active+clean; 170 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 8.2 MiB/s wr, 814 op/s 2024-03-26T01:54:03.062 INFO:teuthology.orchestra.run.smithi019.stdout: 5 20398 17.39 MB/sec warmup 36 sec latency 31.277 ms 2024-03-26T01:54:04.062 INFO:teuthology.orchestra.run.smithi019.stdout: 5 21224 17.89 MB/sec warmup 37 sec latency 22.421 ms 2024-03-26T01:54:04.374 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:04 smithi019 ceph-mon[24816]: pgmap v259: 97 pgs: 97 active+clean; 177 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 9.7 MiB/s wr, 935 op/s 2024-03-26T01:54:04.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:04 smithi069 ceph-mon[27071]: pgmap v259: 97 pgs: 97 active+clean; 177 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 9.7 MiB/s wr, 935 op/s 2024-03-26T01:54:05.062 INFO:teuthology.orchestra.run.smithi019.stdout: 5 21727 17.84 MB/sec warmup 38 sec latency 22.986 ms 2024-03-26T01:54:06.063 INFO:teuthology.orchestra.run.smithi019.stdout: 5 22207 17.50 MB/sec warmup 39 sec latency 32.898 ms 2024-03-26T01:54:06.383 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:06 smithi019 ceph-mon[24816]: pgmap v260: 97 pgs: 97 active+clean; 181 MiB data, 672 MiB used, 715 GiB / 715 GiB avail; 9.9 KiB/s rd, 9.2 MiB/s wr, 912 op/s 2024-03-26T01:54:06.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:06 smithi069 ceph-mon[27071]: pgmap v260: 97 pgs: 97 active+clean; 181 MiB data, 672 MiB used, 715 GiB / 715 GiB avail; 9.9 KiB/s rd, 9.2 MiB/s wr, 912 op/s 2024-03-26T01:54:07.063 INFO:teuthology.orchestra.run.smithi019.stdout: 5 22684 17.38 MB/sec warmup 40 sec latency 55.054 ms 2024-03-26T01:54:08.063 INFO:teuthology.orchestra.run.smithi019.stdout: 5 23341 17.26 MB/sec warmup 41 sec latency 34.861 ms 2024-03-26T01:54:08.405 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:08 smithi019 ceph-mon[24816]: pgmap v261: 97 pgs: 97 active+clean; 180 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 9.7 KiB/s rd, 8.2 MiB/s wr, 816 op/s 2024-03-26T01:54:08.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:08 smithi069 ceph-mon[27071]: pgmap v261: 97 pgs: 97 active+clean; 180 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 9.7 KiB/s rd, 8.2 MiB/s wr, 816 op/s 2024-03-26T01:54:09.063 INFO:teuthology.orchestra.run.smithi019.stdout: 5 23874 17.15 MB/sec warmup 42 sec latency 28.884 ms 2024-03-26T01:54:10.063 INFO:teuthology.orchestra.run.smithi019.stdout: 5 24479 17.37 MB/sec warmup 43 sec latency 37.392 ms 2024-03-26T01:54:10.434 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:10 smithi019 ceph-mon[24816]: pgmap v262: 97 pgs: 97 active+clean; 189 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 9.4 MiB/s wr, 903 op/s 2024-03-26T01:54:10.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:10 smithi069 ceph-mon[27071]: pgmap v262: 97 pgs: 97 active+clean; 189 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 9.4 MiB/s wr, 903 op/s 2024-03-26T01:54:11.063 INFO:teuthology.orchestra.run.smithi019.stdout: 5 25003 17.42 MB/sec warmup 44 sec latency 26.204 ms 2024-03-26T01:54:12.063 INFO:teuthology.orchestra.run.smithi019.stdout: 5 25332 17.21 MB/sec warmup 45 sec latency 46.611 ms 2024-03-26T01:54:12.445 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:12 smithi019 ceph-mon[24816]: pgmap v263: 97 pgs: 97 active+clean; 191 MiB data, 720 MiB used, 715 GiB / 715 GiB avail; 8.3 KiB/s rd, 7.2 MiB/s wr, 659 op/s 2024-03-26T01:54:12.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:12 smithi069 ceph-mon[27071]: pgmap v263: 97 pgs: 97 active+clean; 191 MiB data, 720 MiB used, 715 GiB / 715 GiB avail; 8.3 KiB/s rd, 7.2 MiB/s wr, 659 op/s 2024-03-26T01:54:13.063 INFO:teuthology.orchestra.run.smithi019.stdout: 5 25830 16.95 MB/sec warmup 46 sec latency 31.801 ms 2024-03-26T01:54:14.063 INFO:teuthology.orchestra.run.smithi019.stdout: 5 26425 16.96 MB/sec warmup 47 sec latency 36.451 ms 2024-03-26T01:54:14.467 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:14 smithi019 ceph-mon[24816]: pgmap v264: 97 pgs: 97 active+clean; 197 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 8.5 MiB/s wr, 806 op/s 2024-03-26T01:54:14.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:14 smithi069 ceph-mon[27071]: pgmap v264: 97 pgs: 97 active+clean; 197 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 8.5 MiB/s wr, 806 op/s 2024-03-26T01:54:15.063 INFO:teuthology.orchestra.run.smithi019.stdout: 5 27262 16.91 MB/sec warmup 48 sec latency 29.797 ms 2024-03-26T01:54:16.063 INFO:teuthology.orchestra.run.smithi019.stdout: 5 28084 17.21 MB/sec warmup 49 sec latency 22.767 ms 2024-03-26T01:54:16.227 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:16 smithi019 ceph-mon[24816]: pgmap v265: 97 pgs: 97 active+clean; 203 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 9.7 KiB/s rd, 8.4 MiB/s wr, 793 op/s 2024-03-26T01:54:16.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:16 smithi069 ceph-mon[27071]: pgmap v265: 97 pgs: 97 active+clean; 203 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 9.7 KiB/s rd, 8.4 MiB/s wr, 793 op/s 2024-03-26T01:54:17.063 INFO:teuthology.orchestra.run.smithi019.stdout: 5 28756 17.36 MB/sec warmup 50 sec latency 23.098 ms 2024-03-26T01:54:18.063 INFO:teuthology.orchestra.run.smithi019.stdout: 5 29262 17.13 MB/sec warmup 51 sec latency 29.063 ms 2024-03-26T01:54:18.237 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:18 smithi019 ceph-mon[24816]: pgmap v266: 97 pgs: 97 active+clean; 209 MiB data, 838 MiB used, 714 GiB / 715 GiB avail; 9.6 KiB/s rd, 8.5 MiB/s wr, 837 op/s 2024-03-26T01:54:18.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:18 smithi069 ceph-mon[27071]: pgmap v266: 97 pgs: 97 active+clean; 209 MiB data, 838 MiB used, 714 GiB / 715 GiB avail; 9.6 KiB/s rd, 8.5 MiB/s wr, 837 op/s 2024-03-26T01:54:19.063 INFO:teuthology.orchestra.run.smithi019.stdout: 5 29735 16.99 MB/sec warmup 52 sec latency 35.551 ms 2024-03-26T01:54:19.241 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:19 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:54:19.241 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:19 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/trash_purge_schedule"}]: dispatch 2024-03-26T01:54:19.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:19 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:54:19.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:19 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/trash_purge_schedule"}]: dispatch 2024-03-26T01:54:20.063 INFO:teuthology.orchestra.run.smithi019.stdout: 5 30382 16.95 MB/sec warmup 53 sec latency 90.156 ms 2024-03-26T01:54:20.254 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:20 smithi019 ceph-mon[24816]: pgmap v267: 97 pgs: 97 active+clean; 211 MiB data, 890 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 9.8 MiB/s wr, 876 op/s 2024-03-26T01:54:20.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:20 smithi069 ceph-mon[27071]: pgmap v267: 97 pgs: 97 active+clean; 211 MiB data, 890 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 9.8 MiB/s wr, 876 op/s 2024-03-26T01:54:21.063 INFO:teuthology.orchestra.run.smithi019.stdout: 5 31008 16.90 MB/sec warmup 54 sec latency 50.495 ms 2024-03-26T01:54:21.261 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:21 smithi019 ceph-mon[24816]: pgmap v268: 97 pgs: 97 active+clean; 211 MiB data, 895 MiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 7.5 MiB/s wr, 683 op/s 2024-03-26T01:54:21.544 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:21 smithi069 ceph-mon[27071]: pgmap v268: 97 pgs: 97 active+clean; 211 MiB data, 895 MiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 7.5 MiB/s wr, 683 op/s 2024-03-26T01:54:22.063 INFO:teuthology.orchestra.run.smithi019.stdout: 5 31723 17.14 MB/sec warmup 55 sec latency 22.733 ms 2024-03-26T01:54:23.064 INFO:teuthology.orchestra.run.smithi019.stdout: 5 32346 17.23 MB/sec warmup 56 sec latency 24.874 ms 2024-03-26T01:54:23.912 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:23 smithi019 ceph-mon[24816]: pgmap v269: 97 pgs: 97 active+clean; 218 MiB data, 922 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 9.1 MiB/s wr, 897 op/s 2024-03-26T01:54:24.063 INFO:teuthology.orchestra.run.smithi019.stdout: 5 33030 17.12 MB/sec warmup 57 sec latency 15.492 ms 2024-03-26T01:54:24.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:23 smithi069 ceph-mon[27071]: pgmap v269: 97 pgs: 97 active+clean; 218 MiB data, 922 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 9.1 MiB/s wr, 897 op/s 2024-03-26T01:54:25.063 INFO:teuthology.orchestra.run.smithi019.stdout: 5 33938 17.14 MB/sec warmup 58 sec latency 28.517 ms 2024-03-26T01:54:25.923 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:25 smithi019 ceph-mon[24816]: pgmap v270: 97 pgs: 97 active+clean; 222 MiB data, 950 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.2 MiB/s wr, 863 op/s 2024-03-26T01:54:26.063 INFO:teuthology.orchestra.run.smithi019.stdout: 5 34857 17.32 MB/sec warmup 59 sec latency 21.448 ms 2024-03-26T01:54:26.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:25 smithi069 ceph-mon[27071]: pgmap v270: 97 pgs: 97 active+clean; 222 MiB data, 950 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.2 MiB/s wr, 863 op/s 2024-03-26T01:54:27.064 INFO:teuthology.orchestra.run.smithi019.stdout: 5 35629 17.55 MB/sec warmup 60 sec latency 20.840 ms 2024-03-26T01:54:27.511 INFO:teuthology.orchestra.run.smithi019.stdout:true 2024-03-26T01:54:27.949 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:27 smithi019 ceph-mon[24816]: pgmap v271: 97 pgs: 97 active+clean; 230 MiB data, 989 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.4 MiB/s wr, 947 op/s 2024-03-26T01:54:28.063 INFO:teuthology.orchestra.run.smithi019.stdout: 5 36005 17.40 MB/sec warmup 61 sec latency 39.288 ms 2024-03-26T01:54:28.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:27 smithi069 ceph-mon[27071]: pgmap v271: 97 pgs: 97 active+clean; 230 MiB data, 989 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.4 MiB/s wr, 947 op/s 2024-03-26T01:54:28.365 INFO:teuthology.orchestra.run.smithi019.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T01:54:28.365 INFO:teuthology.orchestra.run.smithi019.stdout:alertmanager.smithi019 smithi019 *:9093,9094 running (6m) 64s ago 7m 23.2M - ba2b418f427c b72e14ab8874 2024-03-26T01:54:28.365 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi019 smithi019 running (7m) 64s ago 7m 6405k - 17.2.0 e1d6a67b021e 713db693d5c1 2024-03-26T01:54:28.365 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi069 smithi069 running (7m) 66s ago 6m 6405k - 17.2.0 e1d6a67b021e a87d27e2b74b 2024-03-26T01:54:28.366 INFO:teuthology.orchestra.run.smithi019.stdout:grafana.smithi019 smithi019 *:3000 running (5m) 64s ago 7m 52.5M - 8.3.5 dad864ee21e9 81e9f452f143 2024-03-26T01:54:28.366 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi019.lmryom smithi019 running (95s) 64s ago 94s 13.2M - 17.2.0 e1d6a67b021e 91a7c2942715 2024-03-26T01:54:28.366 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi069.spgwgo smithi069 running (92s) 66s ago 91s 11.9M - 17.2.0 e1d6a67b021e 94d96338dbe1 2024-03-26T01:54:28.366 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi019.ixuvdm smithi019 *:9283 running (8m) 64s ago 8m 460M - 17.2.0 e1d6a67b021e 1d13a7db0753 2024-03-26T01:54:28.366 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi069.wpbewu smithi069 *:8443,9283 running (6m) 66s ago 6m 412M - 17.2.0 e1d6a67b021e 0a4bd07111bb 2024-03-26T01:54:28.367 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi019 smithi019 running (8m) 64s ago 9m 46.3M 2048M 17.2.0 e1d6a67b021e b6908b497829 2024-03-26T01:54:28.367 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi069 smithi069 running (6m) 66s ago 6m 39.2M 2048M 17.2.0 e1d6a67b021e 2d51b7b7bb32 2024-03-26T01:54:28.367 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.0.0.smithi019.qbizxo smithi019 *:2049 running (74s) 64s ago 74s 60.0M - 4.0 e1d6a67b021e cd6628a9075d 2024-03-26T01:54:28.367 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.1.0.smithi069.uawdhc smithi069 *:2049 running (69s) 66s ago 68s 14.2M - 4.0 e1d6a67b021e 6ee3695e7a10 2024-03-26T01:54:28.367 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi019 smithi019 *:9100 running (7m) 64s ago 7m 17.0M - 1dbe0e931976 8094999d4d38 2024-03-26T01:54:28.367 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi069 smithi069 *:9100 running (6m) 66s ago 6m 17.7M - 1dbe0e931976 46e719c0de82 2024-03-26T01:54:28.368 INFO:teuthology.orchestra.run.smithi019.stdout:osd.0 smithi019 running (6m) 64s ago 6m 43.3M 3217M 17.2.0 e1d6a67b021e c1a797230ab6 2024-03-26T01:54:28.368 INFO:teuthology.orchestra.run.smithi019.stdout:osd.1 smithi019 running (5m) 64s ago 5m 45.5M 3217M 17.2.0 e1d6a67b021e 31d3489c7188 2024-03-26T01:54:28.368 INFO:teuthology.orchestra.run.smithi019.stdout:osd.2 smithi019 running (5m) 64s ago 5m 39.9M 3217M 17.2.0 e1d6a67b021e f549115e7685 2024-03-26T01:54:28.368 INFO:teuthology.orchestra.run.smithi019.stdout:osd.3 smithi019 running (4m) 64s ago 4m 41.6M 3217M 17.2.0 e1d6a67b021e 154af68d03e1 2024-03-26T01:54:28.368 INFO:teuthology.orchestra.run.smithi019.stdout:osd.4 smithi069 running (4m) 66s ago 4m 40.4M 3984M 17.2.0 e1d6a67b021e 4fd5e853f8bf 2024-03-26T01:54:28.369 INFO:teuthology.orchestra.run.smithi019.stdout:osd.5 smithi069 running (3m) 66s ago 3m 38.9M 3984M 17.2.0 e1d6a67b021e 932e401d87c9 2024-03-26T01:54:28.369 INFO:teuthology.orchestra.run.smithi019.stdout:osd.6 smithi069 running (3m) 66s ago 3m 39.3M 3984M 17.2.0 e1d6a67b021e b123f75154c8 2024-03-26T01:54:28.369 INFO:teuthology.orchestra.run.smithi019.stdout:osd.7 smithi069 running (3m) 66s ago 3m 43.6M 3984M 17.2.0 e1d6a67b021e d175fbc9ff8a 2024-03-26T01:54:28.369 INFO:teuthology.orchestra.run.smithi019.stdout:prometheus.smithi019 smithi019 *:9095 running (6m) 64s ago 6m 62.8M - 514e6a882f6e 33689b2df9a5 2024-03-26T01:54:28.861 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T01:54:28.861 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": { 2024-03-26T01:54:28.862 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T01:54:28.862 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:54:28.862 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": { 2024-03-26T01:54:28.862 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T01:54:28.862 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:54:28.862 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": { 2024-03-26T01:54:28.862 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T01:54:28.862 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:54:28.862 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": { 2024-03-26T01:54:28.862 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T01:54:28.862 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:54:28.862 INFO:teuthology.orchestra.run.smithi019.stdout: "overall": { 2024-03-26T01:54:28.863 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-03-26T01:54:28.863 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-26T01:54:28.863 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T01:54:28.958 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:28 smithi019 ceph-mon[24816]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:54:28.958 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:28 smithi019 ceph-mon[24816]: from='client.14630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:54:29.064 INFO:teuthology.orchestra.run.smithi019.stdout: 5 36390 17.18 MB/sec warmup 62 sec latency 40.263 ms 2024-03-26T01:54:29.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:28 smithi069 ceph-mon[27071]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:54:29.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:28 smithi069 ceph-mon[27071]: from='client.14630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:54:29.353 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T01:54:29.353 INFO:teuthology.orchestra.run.smithi019.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a", 2024-03-26T01:54:29.353 INFO:teuthology.orchestra.run.smithi019.stdout: "in_progress": true, 2024-03-26T01:54:29.353 INFO:teuthology.orchestra.run.smithi019.stdout: "services_complete": [], 2024-03-26T01:54:29.353 INFO:teuthology.orchestra.run.smithi019.stdout: "progress": "", 2024-03-26T01:54:29.353 INFO:teuthology.orchestra.run.smithi019.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a image" 2024-03-26T01:54:29.353 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T01:54:29.962 INFO:teuthology.orchestra.run.smithi019.stdout:HEALTH_OK 2024-03-26T01:54:29.965 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:29 smithi019 ceph-mon[24816]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:54:29.965 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:29 smithi019 ceph-mon[24816]: pgmap v272: 97 pgs: 97 active+clean; 234 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 10 MiB/s wr, 862 op/s 2024-03-26T01:54:29.965 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:29 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/2855819566' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T01:54:30.064 INFO:teuthology.orchestra.run.smithi019.stdout: 5 36965 17.20 MB/sec warmup 63 sec latency 83.109 ms 2024-03-26T01:54:30.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:29 smithi069 ceph-mon[27071]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:54:30.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:29 smithi069 ceph-mon[27071]: pgmap v272: 97 pgs: 97 active+clean; 234 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 10 MiB/s wr, 862 op/s 2024-03-26T01:54:30.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:29 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/2855819566' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T01:54:30.983 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:30 smithi019 ceph-mon[24816]: from='client.14642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:54:30.983 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:30 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/3420967983' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T01:54:31.064 INFO:teuthology.orchestra.run.smithi019.stdout: 5 37787 17.08 MB/sec warmup 64 sec latency 43.747 ms 2024-03-26T01:54:31.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:30 smithi069 ceph-mon[27071]: from='client.14642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:54:31.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:30 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/3420967983' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T01:54:32.002 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:31 smithi019 ceph-mon[24816]: pgmap v273: 97 pgs: 97 active+clean; 233 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.2 MiB/s wr, 770 op/s 2024-03-26T01:54:32.064 INFO:teuthology.orchestra.run.smithi019.stdout: 5 38718 17.39 MB/sec warmup 65 sec latency 63.004 ms 2024-03-26T01:54:32.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:31 smithi069 ceph-mon[27071]: pgmap v273: 97 pgs: 97 active+clean; 233 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.2 MiB/s wr, 770 op/s 2024-03-26T01:54:33.064 INFO:teuthology.orchestra.run.smithi019.stdout: 5 39407 17.51 MB/sec warmup 66 sec latency 46.628 ms 2024-03-26T01:54:34.013 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:33 smithi019 ceph-mon[24816]: pgmap v274: 97 pgs: 97 active+clean; 239 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 1.03k op/s 2024-03-26T01:54:34.064 INFO:teuthology.orchestra.run.smithi019.stdout: 5 39963 17.33 MB/sec warmup 67 sec latency 23.247 ms 2024-03-26T01:54:34.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:33 smithi069 ceph-mon[27071]: pgmap v274: 97 pgs: 97 active+clean; 239 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 1.03k op/s 2024-03-26T01:54:35.064 INFO:teuthology.orchestra.run.smithi019.stdout: 5 40621 17.35 MB/sec warmup 68 sec latency 23.375 ms 2024-03-26T01:54:35.272 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:35 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:54:35.272 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:35 smithi019 ceph-mon[24816]: Upgrade: Target is version 19.0.0-1775-g5e70124b (unknown) 2024-03-26T01:54:35.272 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:35 smithi019 ceph-mon[24816]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2'] 2024-03-26T01:54:35.272 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:35 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:54:35.272 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:35 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:54:35.273 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:35 smithi019 ceph-mon[24816]: Upgrade: Need to upgrade myself (mgr.smithi019.ixuvdm) 2024-03-26T01:54:35.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:35 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:54:35.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:35 smithi069 ceph-mon[27071]: Upgrade: Target is version 19.0.0-1775-g5e70124b (unknown) 2024-03-26T01:54:35.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:35 smithi069 ceph-mon[27071]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2'] 2024-03-26T01:54:35.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:35 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:54:35.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:35 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:54:35.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:35 smithi069 ceph-mon[27071]: Upgrade: Need to upgrade myself (mgr.smithi019.ixuvdm) 2024-03-26T01:54:36.064 INFO:teuthology.orchestra.run.smithi019.stdout: 5 41498 17.31 MB/sec warmup 69 sec latency 23.071 ms 2024-03-26T01:54:36.276 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:36 smithi019 ceph-mon[24816]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 on smithi069 2024-03-26T01:54:36.276 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:36 smithi019 ceph-mon[24816]: pgmap v275: 97 pgs: 97 active+clean; 240 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.8 MiB/s wr, 868 op/s 2024-03-26T01:54:36.509 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:36 smithi069 ceph-mon[27071]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 on smithi069 2024-03-26T01:54:36.509 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:36 smithi069 ceph-mon[27071]: pgmap v275: 97 pgs: 97 active+clean; 240 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.8 MiB/s wr, 868 op/s 2024-03-26T01:54:37.064 INFO:teuthology.orchestra.run.smithi019.stdout: 5 42439 17.63 MB/sec warmup 70 sec latency 15.496 ms 2024-03-26T01:54:38.064 INFO:teuthology.orchestra.run.smithi019.stdout: 5 43137 17.64 MB/sec warmup 71 sec latency 30.025 ms 2024-03-26T01:54:38.288 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:38 smithi019 ceph-mon[24816]: pgmap v276: 97 pgs: 97 active+clean; 246 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.5 MiB/s wr, 857 op/s 2024-03-26T01:54:38.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:38 smithi069 ceph-mon[27071]: pgmap v276: 97 pgs: 97 active+clean; 246 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.5 MiB/s wr, 857 op/s 2024-03-26T01:54:39.064 INFO:teuthology.orchestra.run.smithi019.stdout: 5 43749 17.54 MB/sec warmup 72 sec latency 25.002 ms 2024-03-26T01:54:40.064 INFO:teuthology.orchestra.run.smithi019.stdout: 5 44537 17.55 MB/sec warmup 73 sec latency 28.181 ms 2024-03-26T01:54:40.334 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:40 smithi019 ceph-mon[24816]: pgmap v277: 97 pgs: 97 active+clean; 252 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 833 op/s 2024-03-26T01:54:40.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:40 smithi069 ceph-mon[27071]: pgmap v277: 97 pgs: 97 active+clean; 252 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 833 op/s 2024-03-26T01:54:41.064 INFO:teuthology.orchestra.run.smithi019.stdout: 5 45430 17.65 MB/sec warmup 74 sec latency 24.352 ms 2024-03-26T01:54:42.064 INFO:teuthology.orchestra.run.smithi019.stdout: 5 46431 17.93 MB/sec warmup 75 sec latency 21.087 ms 2024-03-26T01:54:42.339 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:42 smithi019 ceph-mon[24816]: pgmap v278: 97 pgs: 97 active+clean; 254 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.5 MiB/s wr, 774 op/s 2024-03-26T01:54:42.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:42 smithi069 ceph-mon[27071]: pgmap v278: 97 pgs: 97 active+clean; 254 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.5 MiB/s wr, 774 op/s 2024-03-26T01:54:43.064 INFO:teuthology.orchestra.run.smithi019.stdout: 5 47173 17.85 MB/sec warmup 76 sec latency 18.740 ms 2024-03-26T01:54:44.064 INFO:teuthology.orchestra.run.smithi019.stdout: 5 48024 17.87 MB/sec warmup 77 sec latency 22.018 ms 2024-03-26T01:54:44.353 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:44 smithi019 ceph-mon[24816]: pgmap v279: 97 pgs: 97 active+clean; 269 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 1.10k op/s 2024-03-26T01:54:44.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:44 smithi069 ceph-mon[27071]: pgmap v279: 97 pgs: 97 active+clean; 269 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 1.10k op/s 2024-03-26T01:54:45.064 INFO:teuthology.orchestra.run.smithi019.stdout: 5 48952 17.95 MB/sec warmup 78 sec latency 22.343 ms 2024-03-26T01:54:46.064 INFO:teuthology.orchestra.run.smithi019.stdout: 5 49793 18.14 MB/sec warmup 79 sec latency 41.040 ms 2024-03-26T01:54:46.463 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:46 smithi019 ceph-mon[24816]: pgmap v280: 97 pgs: 97 active+clean; 270 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 985 op/s 2024-03-26T01:54:46.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:46 smithi069 ceph-mon[27071]: pgmap v280: 97 pgs: 97 active+clean; 270 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 985 op/s 2024-03-26T01:54:47.064 INFO:teuthology.orchestra.run.smithi019.stdout: 5 50240 18.05 MB/sec warmup 80 sec latency 320.597 ms 2024-03-26T01:54:48.064 INFO:teuthology.orchestra.run.smithi019.stdout: 5 50947 17.96 MB/sec warmup 81 sec latency 17.259 ms 2024-03-26T01:54:48.225 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:48 smithi019 ceph-mon[24816]: pgmap v281: 97 pgs: 97 active+clean; 280 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 12 MiB/s wr, 1.17k op/s 2024-03-26T01:54:48.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:48 smithi069 ceph-mon[27071]: pgmap v281: 97 pgs: 97 active+clean; 280 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 12 MiB/s wr, 1.17k op/s 2024-03-26T01:54:49.064 INFO:teuthology.orchestra.run.smithi019.stdout: 5 51819 17.99 MB/sec warmup 82 sec latency 24.098 ms 2024-03-26T01:54:50.064 INFO:teuthology.orchestra.run.smithi019.stdout: 5 52674 18.11 MB/sec warmup 83 sec latency 25.749 ms 2024-03-26T01:54:50.254 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:50 smithi019 ceph-mon[24816]: pgmap v282: 97 pgs: 97 active+clean; 284 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 13 MiB/s wr, 1.17k op/s 2024-03-26T01:54:50.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:50 smithi069 ceph-mon[27071]: pgmap v282: 97 pgs: 97 active+clean; 284 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 13 MiB/s wr, 1.17k op/s 2024-03-26T01:54:51.064 INFO:teuthology.orchestra.run.smithi019.stdout: 5 53499 18.28 MB/sec warmup 84 sec latency 28.032 ms 2024-03-26T01:54:52.064 INFO:teuthology.orchestra.run.smithi019.stdout: 5 54128 18.18 MB/sec warmup 85 sec latency 25.041 ms 2024-03-26T01:54:52.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:52 smithi019 ceph-mon[24816]: pgmap v283: 97 pgs: 97 active+clean; 285 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 1.00k op/s 2024-03-26T01:54:52.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:52 smithi069 ceph-mon[27071]: pgmap v283: 97 pgs: 97 active+clean; 285 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 1.00k op/s 2024-03-26T01:54:53.064 INFO:teuthology.orchestra.run.smithi019.stdout: 5 54859 18.15 MB/sec warmup 86 sec latency 24.282 ms 2024-03-26T01:54:54.064 INFO:teuthology.orchestra.run.smithi019.stdout: 5 55642 18.13 MB/sec warmup 87 sec latency 34.834 ms 2024-03-26T01:54:54.388 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:54 smithi019 ceph-mon[24816]: pgmap v284: 97 pgs: 97 active+clean; 292 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 12 MiB/s wr, 1.18k op/s 2024-03-26T01:54:54.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:54 smithi069 ceph-mon[27071]: pgmap v284: 97 pgs: 97 active+clean; 292 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 12 MiB/s wr, 1.18k op/s 2024-03-26T01:54:55.065 INFO:teuthology.orchestra.run.smithi019.stdout: 5 56435 18.27 MB/sec warmup 88 sec latency 28.183 ms 2024-03-26T01:54:56.065 INFO:teuthology.orchestra.run.smithi019.stdout: 5 57048 18.33 MB/sec warmup 89 sec latency 21.751 ms 2024-03-26T01:54:56.427 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:56 smithi019 ceph-mon[24816]: pgmap v285: 97 pgs: 97 active+clean; 294 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 946 op/s 2024-03-26T01:54:56.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:56 smithi069 ceph-mon[27071]: pgmap v285: 97 pgs: 97 active+clean; 294 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 946 op/s 2024-03-26T01:54:57.065 INFO:teuthology.orchestra.run.smithi019.stdout: 5 57671 18.24 MB/sec warmup 90 sec latency 21.433 ms 2024-03-26T01:54:58.065 INFO:teuthology.orchestra.run.smithi019.stdout: 5 58499 18.24 MB/sec warmup 91 sec latency 44.608 ms 2024-03-26T01:54:58.442 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:54:58 smithi019 ceph-mon[24816]: pgmap v286: 97 pgs: 97 active+clean; 303 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 995 op/s 2024-03-26T01:54:58.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:54:58 smithi069 ceph-mon[27071]: pgmap v286: 97 pgs: 97 active+clean; 303 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 995 op/s 2024-03-26T01:54:59.065 INFO:teuthology.orchestra.run.smithi019.stdout: 5 59311 18.21 MB/sec warmup 92 sec latency 33.000 ms 2024-03-26T01:55:00.065 INFO:teuthology.orchestra.run.smithi019.stdout: 5 60014 18.34 MB/sec warmup 93 sec latency 31.519 ms 2024-03-26T01:55:00.453 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:00 smithi019 ceph-mon[24816]: pgmap v287: 97 pgs: 97 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 995 op/s 2024-03-26T01:55:00.515 INFO:teuthology.orchestra.run.smithi019.stdout:true 2024-03-26T01:55:00.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:00 smithi069 ceph-mon[27071]: pgmap v287: 97 pgs: 97 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 995 op/s 2024-03-26T01:55:01.065 INFO:teuthology.orchestra.run.smithi019.stdout: 5 60655 18.38 MB/sec warmup 94 sec latency 26.448 ms 2024-03-26T01:55:01.547 INFO:teuthology.orchestra.run.smithi019.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T01:55:01.547 INFO:teuthology.orchestra.run.smithi019.stdout:alertmanager.smithi019 smithi019 *:9093,9094 running (6m) 97s ago 8m 23.2M - ba2b418f427c b72e14ab8874 2024-03-26T01:55:01.548 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi019 smithi019 running (8m) 97s ago 8m 6405k - 17.2.0 e1d6a67b021e 713db693d5c1 2024-03-26T01:55:01.548 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi069 smithi069 running (7m) 99s ago 7m 6405k - 17.2.0 e1d6a67b021e a87d27e2b74b 2024-03-26T01:55:01.548 INFO:teuthology.orchestra.run.smithi019.stdout:grafana.smithi019 smithi019 *:3000 running (6m) 97s ago 7m 52.5M - 8.3.5 dad864ee21e9 81e9f452f143 2024-03-26T01:55:01.548 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi019.lmryom smithi019 running (2m) 97s ago 2m 13.2M - 17.2.0 e1d6a67b021e 91a7c2942715 2024-03-26T01:55:01.548 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi069.spgwgo smithi069 running (2m) 99s ago 2m 11.9M - 17.2.0 e1d6a67b021e 94d96338dbe1 2024-03-26T01:55:01.548 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi019.ixuvdm smithi019 *:9283 running (9m) 97s ago 9m 460M - 17.2.0 e1d6a67b021e 1d13a7db0753 2024-03-26T01:55:01.548 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi069.wpbewu smithi069 *:8443,9283 running (7m) 99s ago 7m 412M - 17.2.0 e1d6a67b021e 0a4bd07111bb 2024-03-26T01:55:01.548 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi019 smithi019 running (9m) 97s ago 9m 46.3M 2048M 17.2.0 e1d6a67b021e b6908b497829 2024-03-26T01:55:01.548 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi069 smithi069 running (7m) 99s ago 7m 39.2M 2048M 17.2.0 e1d6a67b021e 2d51b7b7bb32 2024-03-26T01:55:01.548 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.0.0.smithi019.qbizxo smithi019 *:2049 running (107s) 97s ago 107s 60.0M - 4.0 e1d6a67b021e cd6628a9075d 2024-03-26T01:55:01.548 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.1.0.smithi069.uawdhc smithi069 *:2049 running (102s) 99s ago 102s 14.2M - 4.0 e1d6a67b021e 6ee3695e7a10 2024-03-26T01:55:01.548 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi019 smithi019 *:9100 running (7m) 97s ago 7m 17.0M - 1dbe0e931976 8094999d4d38 2024-03-26T01:55:01.548 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi069 smithi069 *:9100 running (7m) 99s ago 7m 17.7M - 1dbe0e931976 46e719c0de82 2024-03-26T01:55:01.549 INFO:teuthology.orchestra.run.smithi019.stdout:osd.0 smithi019 running (6m) 97s ago 6m 43.3M 3217M 17.2.0 e1d6a67b021e c1a797230ab6 2024-03-26T01:55:01.549 INFO:teuthology.orchestra.run.smithi019.stdout:osd.1 smithi019 running (6m) 97s ago 6m 45.5M 3217M 17.2.0 e1d6a67b021e 31d3489c7188 2024-03-26T01:55:01.549 INFO:teuthology.orchestra.run.smithi019.stdout:osd.2 smithi019 running (5m) 97s ago 5m 39.9M 3217M 17.2.0 e1d6a67b021e f549115e7685 2024-03-26T01:55:01.549 INFO:teuthology.orchestra.run.smithi019.stdout:osd.3 smithi019 running (5m) 97s ago 5m 41.6M 3217M 17.2.0 e1d6a67b021e 154af68d03e1 2024-03-26T01:55:01.549 INFO:teuthology.orchestra.run.smithi019.stdout:osd.4 smithi069 running (4m) 99s ago 4m 40.4M 3984M 17.2.0 e1d6a67b021e 4fd5e853f8bf 2024-03-26T01:55:01.549 INFO:teuthology.orchestra.run.smithi019.stdout:osd.5 smithi069 running (4m) 99s ago 4m 38.9M 3984M 17.2.0 e1d6a67b021e 932e401d87c9 2024-03-26T01:55:01.549 INFO:teuthology.orchestra.run.smithi019.stdout:osd.6 smithi069 running (4m) 99s ago 4m 39.3M 3984M 17.2.0 e1d6a67b021e b123f75154c8 2024-03-26T01:55:01.549 INFO:teuthology.orchestra.run.smithi019.stdout:osd.7 smithi069 running (3m) 99s ago 3m 43.6M 3984M 17.2.0 e1d6a67b021e d175fbc9ff8a 2024-03-26T01:55:01.549 INFO:teuthology.orchestra.run.smithi019.stdout:prometheus.smithi019 smithi019 *:9095 running (6m) 97s ago 6m 62.8M - 514e6a882f6e 33689b2df9a5 2024-03-26T01:55:02.065 INFO:teuthology.orchestra.run.smithi019.stdout: 5 61157 18.28 MB/sec warmup 95 sec latency 19.628 ms 2024-03-26T01:55:02.153 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T01:55:02.153 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": { 2024-03-26T01:55:02.153 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T01:55:02.153 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:55:02.153 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": { 2024-03-26T01:55:02.153 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T01:55:02.153 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:55:02.153 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": { 2024-03-26T01:55:02.154 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T01:55:02.154 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:55:02.154 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": { 2024-03-26T01:55:02.154 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T01:55:02.154 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:55:02.154 INFO:teuthology.orchestra.run.smithi019.stdout: "overall": { 2024-03-26T01:55:02.154 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-03-26T01:55:02.154 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-26T01:55:02.154 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T01:55:02.463 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:02 smithi019 ceph-mon[24816]: from='client.14650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:55:02.463 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:02 smithi019 ceph-mon[24816]: pgmap v288: 97 pgs: 97 active+clean; 309 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.0 MiB/s wr, 848 op/s 2024-03-26T01:55:02.463 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:02 smithi019 ceph-mon[24816]: from='client.24413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:55:02.463 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:02 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/1729018582' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T01:55:02.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:02 smithi069 ceph-mon[27071]: from='client.14650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:55:02.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:02 smithi069 ceph-mon[27071]: pgmap v288: 97 pgs: 97 active+clean; 309 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.0 MiB/s wr, 848 op/s 2024-03-26T01:55:02.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:02 smithi069 ceph-mon[27071]: from='client.24413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:55:02.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:02 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/1729018582' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T01:55:02.669 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T01:55:02.670 INFO:teuthology.orchestra.run.smithi019.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2", 2024-03-26T01:55:02.670 INFO:teuthology.orchestra.run.smithi019.stdout: "in_progress": true, 2024-03-26T01:55:02.670 INFO:teuthology.orchestra.run.smithi019.stdout: "services_complete": [], 2024-03-26T01:55:02.670 INFO:teuthology.orchestra.run.smithi019.stdout: "progress": "0/23 daemons upgraded", 2024-03-26T01:55:02.670 INFO:teuthology.orchestra.run.smithi019.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 image on host smithi069" 2024-03-26T01:55:02.670 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T01:55:03.065 INFO:teuthology.orchestra.run.smithi019.stdout: 5 61852 18.24 MB/sec warmup 96 sec latency 24.641 ms 2024-03-26T01:55:03.293 INFO:teuthology.orchestra.run.smithi019.stdout:HEALTH_OK 2024-03-26T01:55:03.472 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:03 smithi019 ceph-mon[24816]: from='client.14658 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:55:03.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:03 smithi069 ceph-mon[27071]: from='client.14658 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:55:04.065 INFO:teuthology.orchestra.run.smithi019.stdout: 5 62677 18.23 MB/sec warmup 97 sec latency 23.748 ms 2024-03-26T01:55:04.485 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:04 smithi019 ceph-mon[24816]: from='client.14666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:55:04.485 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:04 smithi019 ceph-mon[24816]: pgmap v289: 97 pgs: 97 active+clean; 313 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 1.02k op/s 2024-03-26T01:55:04.485 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:04 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/3467074107' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T01:55:04.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:04 smithi069 ceph-mon[27071]: from='client.14666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:55:04.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:04 smithi069 ceph-mon[27071]: pgmap v289: 97 pgs: 97 active+clean; 313 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 1.02k op/s 2024-03-26T01:55:04.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:04 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/3467074107' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T01:55:05.065 INFO:teuthology.orchestra.run.smithi019.stdout: 5 63291 18.26 MB/sec warmup 98 sec latency 25.205 ms 2024-03-26T01:55:06.065 INFO:teuthology.orchestra.run.smithi019.stdout: 5 64051 18.37 MB/sec warmup 99 sec latency 22.021 ms 2024-03-26T01:55:06.250 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:06 smithi019 ceph-mon[24816]: pgmap v290: 97 pgs: 97 active+clean; 317 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 893 op/s 2024-03-26T01:55:06.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:06 smithi069 ceph-mon[27071]: pgmap v290: 97 pgs: 97 active+clean; 317 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 893 op/s 2024-03-26T01:55:07.065 INFO:teuthology.orchestra.run.smithi019.stdout: 5 64463 18.28 MB/sec warmup 100 sec latency 25.896 ms 2024-03-26T01:55:07.259 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:07 smithi019 ceph-mon[24816]: pgmap v291: 97 pgs: 97 active+clean; 320 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.8 MiB/s wr, 909 op/s 2024-03-26T01:55:07.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:07 smithi069 ceph-mon[27071]: pgmap v291: 97 pgs: 97 active+clean; 320 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.8 MiB/s wr, 909 op/s 2024-03-26T01:55:08.065 INFO:teuthology.orchestra.run.smithi019.stdout: 5 64992 18.18 MB/sec warmup 101 sec latency 60.251 ms 2024-03-26T01:55:09.065 INFO:teuthology.orchestra.run.smithi019.stdout: 5 65571 18.14 MB/sec warmup 102 sec latency 39.212 ms 2024-03-26T01:55:09.810 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:09 smithi019 ceph-mon[24816]: pgmap v292: 97 pgs: 97 active+clean; 329 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-03-26T01:55:10.065 INFO:teuthology.orchestra.run.smithi019.stdout: 5 66313 18.10 MB/sec warmup 103 sec latency 33.139 ms 2024-03-26T01:55:10.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:09 smithi069 ceph-mon[27071]: pgmap v292: 97 pgs: 97 active+clean; 329 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-03-26T01:55:11.065 INFO:teuthology.orchestra.run.smithi019.stdout: 5 66863 18.15 MB/sec warmup 104 sec latency 29.390 ms 2024-03-26T01:55:11.869 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:11 smithi019 ceph-mon[24816]: pgmap v293: 97 pgs: 97 active+clean; 330 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.3 MiB/s wr, 789 op/s 2024-03-26T01:55:12.065 INFO:teuthology.orchestra.run.smithi019.stdout: 5 67492 18.19 MB/sec warmup 105 sec latency 41.346 ms 2024-03-26T01:55:12.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:11 smithi069 ceph-mon[27071]: pgmap v293: 97 pgs: 97 active+clean; 330 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.3 MiB/s wr, 789 op/s 2024-03-26T01:55:13.065 INFO:teuthology.orchestra.run.smithi019.stdout: 5 67910 18.13 MB/sec warmup 106 sec latency 26.645 ms 2024-03-26T01:55:13.894 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:13 smithi019 ceph-mon[24816]: pgmap v294: 97 pgs: 97 active+clean; 328 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 17 KiB/s rd, 9.3 MiB/s wr, 840 op/s 2024-03-26T01:55:13.894 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:13 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:55:13.894 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:13 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi069.wpbewu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T01:55:13.894 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:13 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T01:55:13.894 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:13 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:55:14.065 INFO:teuthology.orchestra.run.smithi019.stdout: 5 68332 18.02 MB/sec warmup 107 sec latency 95.909 ms 2024-03-26T01:55:14.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:13 smithi069 ceph-mon[27071]: pgmap v294: 97 pgs: 97 active+clean; 328 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 17 KiB/s rd, 9.3 MiB/s wr, 840 op/s 2024-03-26T01:55:14.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:13 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:55:14.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:13 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi069.wpbewu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T01:55:14.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:13 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T01:55:14.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:13 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:55:14.845 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:14 smithi019 ceph-mon[24816]: Upgrade: Updating mgr.smithi069.wpbewu 2024-03-26T01:55:14.845 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:14 smithi019 ceph-mon[24816]: Deploying daemon mgr.smithi069.wpbewu on smithi069 2024-03-26T01:55:15.065 INFO:teuthology.orchestra.run.smithi019.stdout: 5 68770 17.94 MB/sec warmup 108 sec latency 37.638 ms 2024-03-26T01:55:15.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:14 smithi069 ceph-mon[27071]: Upgrade: Updating mgr.smithi069.wpbewu 2024-03-26T01:55:15.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:14 smithi069 ceph-mon[27071]: Deploying daemon mgr.smithi069.wpbewu on smithi069 2024-03-26T01:55:15.914 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:15 smithi019 ceph-mon[24816]: pgmap v295: 97 pgs: 97 active+clean; 332 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.8 MiB/s wr, 784 op/s 2024-03-26T01:55:16.065 INFO:teuthology.orchestra.run.smithi019.stdout: 5 69276 17.89 MB/sec warmup 109 sec latency 43.082 ms 2024-03-26T01:55:16.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:15 smithi069 ceph-mon[27071]: pgmap v295: 97 pgs: 97 active+clean; 332 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.8 MiB/s wr, 784 op/s 2024-03-26T01:55:17.065 INFO:teuthology.orchestra.run.smithi019.stdout: 5 69808 17.81 MB/sec warmup 110 sec latency 54.437 ms 2024-03-26T01:55:17.966 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:17 smithi019 ceph-mon[24816]: pgmap v296: 97 pgs: 97 active+clean; 342 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 9.0 MiB/s wr, 941 op/s 2024-03-26T01:55:18.065 INFO:teuthology.orchestra.run.smithi019.stdout: 5 70272 17.82 MB/sec warmup 111 sec latency 29.689 ms 2024-03-26T01:55:18.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:17 smithi069 ceph-mon[27071]: pgmap v296: 97 pgs: 97 active+clean; 342 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 9.0 MiB/s wr, 941 op/s 2024-03-26T01:55:18.979 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:18 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:55:18.979 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:18 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/trash_purge_schedule"}]: dispatch 2024-03-26T01:55:18.992 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:18 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:55:18.992 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:18 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/trash_purge_schedule"}]: dispatch 2024-03-26T01:55:19.065 INFO:teuthology.orchestra.run.smithi019.stdout: 5 70748 17.83 MB/sec warmup 112 sec latency 38.221 ms 2024-03-26T01:55:19.988 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:19 smithi019 ceph-mon[24816]: pgmap v297: 97 pgs: 97 active+clean; 336 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 8.9 MiB/s wr, 924 op/s 2024-03-26T01:55:19.988 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:19 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:55:19.988 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:19 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:55:19.989 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:19 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:55:19.989 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:19 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:55:20.010 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:19 smithi069 ceph-mon[27071]: pgmap v297: 97 pgs: 97 active+clean; 336 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 8.9 MiB/s wr, 924 op/s 2024-03-26T01:55:20.010 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:19 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:55:20.010 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:19 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:55:20.010 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:19 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:55:20.010 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:19 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:55:20.066 INFO:teuthology.orchestra.run.smithi019.stdout: 5 71332 17.84 MB/sec warmup 113 sec latency 29.631 ms 2024-03-26T01:55:21.065 INFO:teuthology.orchestra.run.smithi019.stdout: 5 71827 17.73 MB/sec warmup 114 sec latency 22.331 ms 2024-03-26T01:55:22.010 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:21 smithi019 ceph-mon[24816]: pgmap v298: 97 pgs: 97 active+clean; 337 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 6.2 MiB/s wr, 636 op/s 2024-03-26T01:55:22.071 INFO:teuthology.orchestra.run.smithi019.stdout: 5 72488 17.71 MB/sec warmup 115 sec latency 36.188 ms 2024-03-26T01:55:22.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:21 smithi069 ceph-mon[27071]: pgmap v298: 97 pgs: 97 active+clean; 337 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 6.2 MiB/s wr, 636 op/s 2024-03-26T01:55:23.073 INFO:teuthology.orchestra.run.smithi019.stdout: 5 73164 17.66 MB/sec warmup 116 sec latency 46.832 ms 2024-03-26T01:55:24.067 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:24 smithi019 ceph-mon[24816]: pgmap v299: 97 pgs: 97 active+clean; 333 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 7.8 MiB/s wr, 820 op/s 2024-03-26T01:55:24.067 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:24 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:55:24.067 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:24 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:55:24.071 INFO:teuthology.orchestra.run.smithi019.stdout: 5 73616 17.61 MB/sec warmup 117 sec latency 38.459 ms 2024-03-26T01:55:24.319 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:24 smithi069 ceph-mon[27071]: pgmap v299: 97 pgs: 97 active+clean; 333 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 7.8 MiB/s wr, 820 op/s 2024-03-26T01:55:24.319 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:24 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:55:24.319 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:24 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:55:25.072 INFO:teuthology.orchestra.run.smithi019.stdout: 5 74193 17.69 MB/sec warmup 118 sec latency 25.412 ms 2024-03-26T01:55:26.072 INFO:teuthology.orchestra.run.smithi019.stdout: 5 74618 17.66 MB/sec warmup 119 sec latency 40.542 ms 2024-03-26T01:55:26.093 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:26 smithi019 ceph-mon[24816]: pgmap v300: 97 pgs: 97 active+clean; 332 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 7.7 MiB/s wr, 794 op/s 2024-03-26T01:55:26.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:26 smithi069 ceph-mon[27071]: pgmap v300: 97 pgs: 97 active+clean; 332 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 7.7 MiB/s wr, 794 op/s 2024-03-26T01:55:28.072 INFO:teuthology.orchestra.run.smithi019.stdout: 5 75398 3.92 MB/sec execute 1 sec latency 36.710 ms 2024-03-26T01:55:28.134 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:28 smithi019 ceph-mon[24816]: pgmap v301: 97 pgs: 97 active+clean; 339 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 7.7 MiB/s wr, 886 op/s 2024-03-26T01:55:28.134 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:28 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:55:28.134 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:28 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:55:28.134 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:28 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:55:28.134 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:28 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:55:28.135 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:28 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mgr fail", "who": "smithi019.ixuvdm"}]: dispatch 2024-03-26T01:55:28.135 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:28 smithi019 ceph-mon[24816]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T01:55:28.135 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:28 smithi019 ceph-mon[24816]: Active manager daemon smithi069.wpbewu restarted 2024-03-26T01:55:28.135 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:28 smithi019 ceph-mon[24816]: from='mgr.? 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi069.wpbewu/crt"}]: dispatch 2024-03-26T01:55:28.135 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:28 smithi019 ceph-mon[24816]: Activating manager daemon smithi069.wpbewu 2024-03-26T01:55:28.135 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:28 smithi019 ceph-mon[24816]: from='mgr.? 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T01:55:28.135 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:28 smithi019 ceph-mon[24816]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T01:55:28.135 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:28 smithi019 ceph-mon[24816]: from='mgr.? 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi069.wpbewu/key"}]: dispatch 2024-03-26T01:55:28.135 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:28 smithi019 ceph-mon[24816]: from='mgr.? 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T01:55:28.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:28 smithi069 ceph-mon[27071]: pgmap v301: 97 pgs: 97 active+clean; 339 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 7.7 MiB/s wr, 886 op/s 2024-03-26T01:55:28.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:28 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:55:28.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:28 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:55:28.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:28 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:55:28.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:28 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:55:28.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:28 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mgr fail", "who": "smithi019.ixuvdm"}]: dispatch 2024-03-26T01:55:28.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:28 smithi069 ceph-mon[27071]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T01:55:28.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:28 smithi069 ceph-mon[27071]: Active manager daemon smithi069.wpbewu restarted 2024-03-26T01:55:28.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:28 smithi069 ceph-mon[27071]: from='mgr.? 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi069.wpbewu/crt"}]: dispatch 2024-03-26T01:55:28.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:28 smithi069 ceph-mon[27071]: Activating manager daemon smithi069.wpbewu 2024-03-26T01:55:28.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:28 smithi069 ceph-mon[27071]: from='mgr.? 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T01:55:28.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:28 smithi069 ceph-mon[27071]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T01:55:28.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:28 smithi069 ceph-mon[27071]: from='mgr.? 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi069.wpbewu/key"}]: dispatch 2024-03-26T01:55:28.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:28 smithi069 ceph-mon[27071]: from='mgr.? 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T01:55:29.072 INFO:teuthology.orchestra.run.smithi019.stdout: 5 75925 8.54 MB/sec execute 2 sec latency 41.407 ms 2024-03-26T01:55:29.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:29 smithi069 ceph-mon[27071]: Upgrade: Need to upgrade myself (mgr.smithi019.ixuvdm) 2024-03-26T01:55:29.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:29 smithi069 ceph-mon[27071]: Failing over to other MGR 2024-03-26T01:55:29.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:29 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "mon metadata", "id": "smithi019"}]: dispatch 2024-03-26T01:55:29.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:29 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "mon metadata", "id": "smithi069"}]: dispatch 2024-03-26T01:55:29.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:29 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi019.lmryom"}]: dispatch 2024-03-26T01:55:29.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:29 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi069.spgwgo"}]: dispatch 2024-03-26T01:55:29.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:29 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "mgr metadata", "who": "smithi069.wpbewu", "id": "smithi069.wpbewu"}]: dispatch 2024-03-26T01:55:29.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:29 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:55:29.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:29 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:55:29.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:29 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:55:29.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:29 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:55:29.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:29 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T01:55:29.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:29 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T01:55:29.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:29 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T01:55:29.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:29 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T01:55:29.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:29 smithi069 ceph-mon[27071]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "mgr fail", "who": "smithi019.ixuvdm"}]': finished 2024-03-26T01:55:29.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:29 smithi069 ceph-mon[27071]: mgrmap e22: smithi069.wpbewu(active, starting, since 0.930087s) 2024-03-26T01:55:29.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:29 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T01:55:29.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:29 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T01:55:29.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:29 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T01:55:29.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:29 smithi069 ceph-mon[27071]: Manager daemon smithi069.wpbewu is now available 2024-03-26T01:55:29.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:29 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:29.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:29 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:55:29.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:29 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:55:29.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:29 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:55:29.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:29 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi069.wpbewu/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:55:29.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:29 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi069.wpbewu/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:55:29.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:29 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi069.wpbewu/trash_purge_schedule"}]: dispatch 2024-03-26T01:55:29.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:29 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi069.wpbewu/trash_purge_schedule"}]: dispatch 2024-03-26T01:55:29.159 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:29 smithi019 ceph-mon[24816]: Upgrade: Need to upgrade myself (mgr.smithi019.ixuvdm) 2024-03-26T01:55:29.160 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:29 smithi019 ceph-mon[24816]: Failing over to other MGR 2024-03-26T01:55:29.160 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:29 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "mon metadata", "id": "smithi019"}]: dispatch 2024-03-26T01:55:29.160 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:29 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "mon metadata", "id": "smithi069"}]: dispatch 2024-03-26T01:55:29.160 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:29 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi019.lmryom"}]: dispatch 2024-03-26T01:55:29.160 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:29 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi069.spgwgo"}]: dispatch 2024-03-26T01:55:29.160 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:29 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "mgr metadata", "who": "smithi069.wpbewu", "id": "smithi069.wpbewu"}]: dispatch 2024-03-26T01:55:29.160 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:29 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:55:29.160 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:29 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:55:29.160 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:29 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:55:29.160 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:29 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:55:29.160 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:29 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T01:55:29.160 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:29 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T01:55:29.160 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:29 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T01:55:29.161 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:29 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T01:55:29.161 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:29 smithi019 ceph-mon[24816]: from='mgr.14204 172.21.15.19:0/3541343918' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "mgr fail", "who": "smithi019.ixuvdm"}]': finished 2024-03-26T01:55:29.161 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:29 smithi019 ceph-mon[24816]: mgrmap e22: smithi069.wpbewu(active, starting, since 0.930087s) 2024-03-26T01:55:29.161 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:29 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T01:55:29.161 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:29 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T01:55:29.161 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:29 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T01:55:29.161 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:29 smithi019 ceph-mon[24816]: Manager daemon smithi069.wpbewu is now available 2024-03-26T01:55:29.161 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:29 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:29.161 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:29 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:55:29.161 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:29 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:55:29.161 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:29 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:55:29.161 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:29 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi069.wpbewu/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:55:29.161 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:29 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi069.wpbewu/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:55:29.161 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:29 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi069.wpbewu/trash_purge_schedule"}]: dispatch 2024-03-26T01:55:29.161 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:29 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi069.wpbewu/trash_purge_schedule"}]: dispatch 2024-03-26T01:55:30.072 INFO:teuthology.orchestra.run.smithi019.stdout: 5 76502 9.69 MB/sec execute 3 sec latency 35.877 ms 2024-03-26T01:55:30.416 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:30 smithi019 ceph-mon[24816]: Deploying cephadm binary to smithi069 2024-03-26T01:55:30.416 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:30 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:30.416 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:30 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:30.416 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:30 smithi019 ceph-mon[24816]: mgrmap e23: smithi069.wpbewu(active, since 1.93108s) 2024-03-26T01:55:30.416 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:30 smithi019 ceph-mon[24816]: pgmap v3: 97 pgs: 97 active+clean; 328 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-03-26T01:55:30.752 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:30 smithi069 ceph-mon[27071]: Deploying cephadm binary to smithi069 2024-03-26T01:55:30.752 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:30 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:30.752 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:30 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:30.752 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:30 smithi069 ceph-mon[27071]: mgrmap e23: smithi069.wpbewu(active, since 1.93108s) 2024-03-26T01:55:30.752 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:30 smithi069 ceph-mon[27071]: pgmap v3: 97 pgs: 97 active+clean; 328 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-03-26T01:55:31.072 INFO:teuthology.orchestra.run.smithi019.stdout: 5 77015 9.98 MB/sec execute 4 sec latency 31.957 ms 2024-03-26T01:55:31.680 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:31 smithi019 ceph-mon[24816]: pgmap v4: 97 pgs: 97 active+clean; 328 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-03-26T01:55:31.680 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:31 smithi019 ceph-mon[24816]: mgrmap e24: smithi069.wpbewu(active, since 2s) 2024-03-26T01:55:31.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:31 smithi069 ceph-mon[27071]: pgmap v4: 97 pgs: 97 active+clean; 328 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-03-26T01:55:31.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:31 smithi069 ceph-mon[27071]: mgrmap e24: smithi069.wpbewu(active, since 2s) 2024-03-26T01:55:32.072 INFO:teuthology.orchestra.run.smithi019.stdout: 5 77431 11.05 MB/sec execute 5 sec latency 52.695 ms 2024-03-26T01:55:32.705 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:32 smithi019 ceph-mon[24816]: [26/Mar/2024:01:55:31] ENGINE Bus STARTING 2024-03-26T01:55:32.705 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:32 smithi019 ceph-mon[24816]: [26/Mar/2024:01:55:31] ENGINE Serving on http://172.21.15.69:8765 2024-03-26T01:55:32.705 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:32 smithi019 ceph-mon[24816]: [26/Mar/2024:01:55:31] ENGINE Serving on https://172.21.15.69:7150 2024-03-26T01:55:32.705 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:32 smithi019 ceph-mon[24816]: [26/Mar/2024:01:55:31] ENGINE Bus STARTED 2024-03-26T01:55:33.077 INFO:teuthology.orchestra.run.smithi019.stdout: 5 78084 13.86 MB/sec execute 6 sec latency 21.687 ms 2024-03-26T01:55:33.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:32 smithi069 ceph-mon[27071]: [26/Mar/2024:01:55:31] ENGINE Bus STARTING 2024-03-26T01:55:33.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:32 smithi069 ceph-mon[27071]: [26/Mar/2024:01:55:31] ENGINE Serving on http://172.21.15.69:8765 2024-03-26T01:55:33.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:32 smithi069 ceph-mon[27071]: [26/Mar/2024:01:55:31] ENGINE Serving on https://172.21.15.69:7150 2024-03-26T01:55:33.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:32 smithi069 ceph-mon[27071]: [26/Mar/2024:01:55:31] ENGINE Bus STARTED 2024-03-26T01:55:33.794 INFO:teuthology.orchestra.run.smithi019.stdout:true 2024-03-26T01:55:33.840 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:33 smithi019 ceph-mon[24816]: Deploying cephadm binary to smithi019 2024-03-26T01:55:33.840 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:33 smithi019 ceph-mon[24816]: pgmap v5: 97 pgs: 97 active+clean; 328 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-03-26T01:55:33.840 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:33 smithi019 ceph-mon[24816]: mgrmap e25: smithi069.wpbewu(active, since 4s) 2024-03-26T01:55:34.078 INFO:teuthology.orchestra.run.smithi019.stdout: 5 78455 13.57 MB/sec execute 7 sec latency 131.441 ms 2024-03-26T01:55:34.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:33 smithi069 ceph-mon[27071]: Deploying cephadm binary to smithi019 2024-03-26T01:55:34.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:33 smithi069 ceph-mon[27071]: pgmap v5: 97 pgs: 97 active+clean; 328 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-03-26T01:55:34.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:33 smithi069 ceph-mon[27071]: mgrmap e25: smithi069.wpbewu(active, since 4s) 2024-03-26T01:55:34.844 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:34 smithi019 ceph-mon[24816]: from='client.24463 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:55:34.899 INFO:teuthology.orchestra.run.smithi019.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T01:55:34.899 INFO:teuthology.orchestra.run.smithi019.stdout:alertmanager.smithi019 smithi019 *:9093,9094 running (7m) 2m ago 8m 23.2M - ba2b418f427c b72e14ab8874 2024-03-26T01:55:34.899 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi019 smithi019 running (8m) 2m ago 8m 6405k - 17.2.0 e1d6a67b021e 713db693d5c1 2024-03-26T01:55:34.899 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi069 smithi069 running (8m) 11s ago 8m 6405k - 17.2.0 e1d6a67b021e a87d27e2b74b 2024-03-26T01:55:34.899 INFO:teuthology.orchestra.run.smithi019.stdout:grafana.smithi019 smithi019 *:3000 running (7m) 2m ago 8m 52.5M - 8.3.5 dad864ee21e9 81e9f452f143 2024-03-26T01:55:34.899 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi019.lmryom smithi019 running (2m) 2m ago 2m 13.2M - 17.2.0 e1d6a67b021e 91a7c2942715 2024-03-26T01:55:34.899 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi069.spgwgo smithi069 running (2m) 11s ago 2m 12.4M - 17.2.0 e1d6a67b021e 94d96338dbe1 2024-03-26T01:55:34.899 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi019.ixuvdm smithi019 *:9283 running (9m) 2m ago 9m 460M - 17.2.0 e1d6a67b021e 1d13a7db0753 2024-03-26T01:55:34.899 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi069.wpbewu smithi069 *:8443,9283 running (17s) 11s ago 8m 46.7M - 19.0.0-1775-g5e70124b 5f21c9264136 c0a5be7a9f9e 2024-03-26T01:55:34.899 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi019 smithi019 running (10m) 2m ago 10m 46.3M 2048M 17.2.0 e1d6a67b021e b6908b497829 2024-03-26T01:55:34.900 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi069 smithi069 running (7m) 11s ago 7m 40.4M 2048M 17.2.0 e1d6a67b021e 2d51b7b7bb32 2024-03-26T01:55:34.900 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.0.0.smithi019.qbizxo smithi019 *:2049 running (2m) 2m ago 2m 60.0M - 4.0 e1d6a67b021e cd6628a9075d 2024-03-26T01:55:34.900 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.1.0.smithi069.uawdhc smithi069 *:2049 running (2m) 11s ago 2m 71.0M - 4.0 e1d6a67b021e 6ee3695e7a10 2024-03-26T01:55:34.900 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi019 smithi019 *:9100 running (8m) 2m ago 8m 17.0M - 1dbe0e931976 8094999d4d38 2024-03-26T01:55:34.900 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi069 smithi069 *:9100 running (7m) 11s ago 7m 19.1M - 1dbe0e931976 46e719c0de82 2024-03-26T01:55:34.900 INFO:teuthology.orchestra.run.smithi019.stdout:osd.0 smithi019 running (7m) 2m ago 7m 43.3M 3217M 17.2.0 e1d6a67b021e c1a797230ab6 2024-03-26T01:55:34.900 INFO:teuthology.orchestra.run.smithi019.stdout:osd.1 smithi019 running (6m) 2m ago 6m 45.5M 3217M 17.2.0 e1d6a67b021e 31d3489c7188 2024-03-26T01:55:34.900 INFO:teuthology.orchestra.run.smithi019.stdout:osd.2 smithi019 running (6m) 2m ago 6m 39.9M 3217M 17.2.0 e1d6a67b021e f549115e7685 2024-03-26T01:55:34.900 INFO:teuthology.orchestra.run.smithi019.stdout:osd.3 smithi019 running (5m) 2m ago 5m 41.6M 3217M 17.2.0 e1d6a67b021e 154af68d03e1 2024-03-26T01:55:34.900 INFO:teuthology.orchestra.run.smithi019.stdout:osd.4 smithi069 running (5m) 11s ago 5m 252M 3984M 17.2.0 e1d6a67b021e 4fd5e853f8bf 2024-03-26T01:55:34.900 INFO:teuthology.orchestra.run.smithi019.stdout:osd.5 smithi069 running (4m) 11s ago 4m 293M 3984M 17.2.0 e1d6a67b021e 932e401d87c9 2024-03-26T01:55:34.900 INFO:teuthology.orchestra.run.smithi019.stdout:osd.6 smithi069 running (4m) 11s ago 4m 270M 3984M 17.2.0 e1d6a67b021e b123f75154c8 2024-03-26T01:55:34.900 INFO:teuthology.orchestra.run.smithi019.stdout:osd.7 smithi069 running (4m) 11s ago 4m 373M 3984M 17.2.0 e1d6a67b021e d175fbc9ff8a 2024-03-26T01:55:34.900 INFO:teuthology.orchestra.run.smithi019.stdout:prometheus.smithi019 smithi019 *:9095 running (7m) 2m ago 7m 62.8M - 514e6a882f6e 33689b2df9a5 2024-03-26T01:55:35.080 INFO:teuthology.orchestra.run.smithi019.stdout: 5 78860 12.34 MB/sec execute 8 sec latency 119.617 ms 2024-03-26T01:55:35.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:34 smithi069 ceph-mon[27071]: from='client.24463 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:55:35.648 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T01:55:35.648 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": { 2024-03-26T01:55:35.648 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T01:55:35.648 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:55:35.648 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": { 2024-03-26T01:55:35.648 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 1 2024-03-26T01:55:35.649 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:55:35.649 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": { 2024-03-26T01:55:35.649 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T01:55:35.649 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:55:35.649 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": { 2024-03-26T01:55:35.649 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T01:55:35.649 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:55:35.649 INFO:teuthology.orchestra.run.smithi019.stdout: "overall": { 2024-03-26T01:55:35.649 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-03-26T01:55:35.649 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 1 2024-03-26T01:55:35.649 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-26T01:55:35.649 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T01:55:35.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:35 smithi069 ceph-mon[27071]: from='client.14698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:55:35.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:35 smithi069 ceph-mon[27071]: pgmap v6: 97 pgs: 97 active+clean; 328 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-03-26T01:55:35.875 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:35 smithi019 ceph-mon[24816]: from='client.14698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:55:35.875 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:35 smithi019 ceph-mon[24816]: pgmap v6: 97 pgs: 97 active+clean; 328 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-03-26T01:55:36.080 INFO:teuthology.orchestra.run.smithi019.stdout: 5 79284 11.91 MB/sec execute 9 sec latency 33.359 ms 2024-03-26T01:55:36.257 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T01:55:36.257 INFO:teuthology.orchestra.run.smithi019.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2", 2024-03-26T01:55:36.258 INFO:teuthology.orchestra.run.smithi019.stdout: "in_progress": true, 2024-03-26T01:55:36.258 INFO:teuthology.orchestra.run.smithi019.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T01:55:36.258 INFO:teuthology.orchestra.run.smithi019.stdout: "services_complete": [], 2024-03-26T01:55:36.258 INFO:teuthology.orchestra.run.smithi019.stdout: "progress": "1/23 daemons upgraded", 2024-03-26T01:55:36.258 INFO:teuthology.orchestra.run.smithi019.stdout: "message": "", 2024-03-26T01:55:36.258 INFO:teuthology.orchestra.run.smithi019.stdout: "is_paused": false 2024-03-26T01:55:36.258 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T01:55:36.759 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:36 smithi069 ceph-mon[27071]: from='client.24471 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:55:36.759 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:36 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/4155856366' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T01:55:36.880 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:36 smithi019 ceph-mon[24816]: from='client.24471 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:55:36.880 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:36 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/4155856366' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T01:55:36.956 INFO:teuthology.orchestra.run.smithi019.stdout:HEALTH_OK 2024-03-26T01:55:37.080 INFO:teuthology.orchestra.run.smithi019.stdout: 5 79789 12.23 MB/sec execute 10 sec latency 38.372 ms 2024-03-26T01:55:37.888 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:37 smithi019 ceph-mon[24816]: from='client.24479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:55:37.888 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:37 smithi019 ceph-mon[24816]: pgmap v7: 97 pgs: 97 active+clean; 329 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 37 KiB/s rd, 5.5 MiB/s wr, 497 op/s 2024-03-26T01:55:37.888 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:37 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/4056259982' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T01:55:38.057 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:37 smithi069 ceph-mon[27071]: from='client.24479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:55:38.057 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:37 smithi069 ceph-mon[27071]: pgmap v7: 97 pgs: 97 active+clean; 329 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 37 KiB/s rd, 5.5 MiB/s wr, 497 op/s 2024-03-26T01:55:38.057 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:37 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/4056259982' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T01:55:38.080 INFO:teuthology.orchestra.run.smithi019.stdout: 5 80368 11.72 MB/sec execute 11 sec latency 36.496 ms 2024-03-26T01:55:38.914 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:38 smithi019 ceph-mon[24816]: Standby manager daemon smithi019.ixuvdm started 2024-03-26T01:55:38.914 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:38 smithi019 ceph-mon[24816]: from='mgr.? 172.21.15.19:0/3810396891' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi019.ixuvdm/crt"}]: dispatch 2024-03-26T01:55:38.914 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:38 smithi019 ceph-mon[24816]: from='mgr.? 172.21.15.19:0/3810396891' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T01:55:38.914 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:38 smithi019 ceph-mon[24816]: from='mgr.? 172.21.15.19:0/3810396891' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi019.ixuvdm/key"}]: dispatch 2024-03-26T01:55:38.914 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:38 smithi019 ceph-mon[24816]: from='mgr.? 172.21.15.19:0/3810396891' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T01:55:38.914 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:38 smithi019 ceph-mon[24816]: pgmap v8: 97 pgs: 97 active+clean; 329 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 4.3 MiB/s wr, 386 op/s 2024-03-26T01:55:39.080 INFO:teuthology.orchestra.run.smithi019.stdout: 5 80944 12.50 MB/sec execute 12 sec latency 23.300 ms 2024-03-26T01:55:39.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:38 smithi069 ceph-mon[27071]: Standby manager daemon smithi019.ixuvdm started 2024-03-26T01:55:39.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:38 smithi069 ceph-mon[27071]: from='mgr.? 172.21.15.19:0/3810396891' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi019.ixuvdm/crt"}]: dispatch 2024-03-26T01:55:39.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:38 smithi069 ceph-mon[27071]: from='mgr.? 172.21.15.19:0/3810396891' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T01:55:39.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:38 smithi069 ceph-mon[27071]: from='mgr.? 172.21.15.19:0/3810396891' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi019.ixuvdm/key"}]: dispatch 2024-03-26T01:55:39.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:38 smithi069 ceph-mon[27071]: from='mgr.? 172.21.15.19:0/3810396891' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T01:55:39.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:38 smithi069 ceph-mon[27071]: pgmap v8: 97 pgs: 97 active+clean; 329 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 4.3 MiB/s wr, 386 op/s 2024-03-26T01:55:39.930 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:39 smithi019 ceph-mon[24816]: mgrmap e26: smithi069.wpbewu(active, since 11s), standbys: smithi019.ixuvdm 2024-03-26T01:55:39.930 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:39 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "mgr metadata", "who": "smithi019.ixuvdm", "id": "smithi019.ixuvdm"}]: dispatch 2024-03-26T01:55:39.930 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:39 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:39.930 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:39 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:40.080 INFO:teuthology.orchestra.run.smithi019.stdout: 5 81542 13.38 MB/sec execute 13 sec latency 46.033 ms 2024-03-26T01:55:40.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:39 smithi069 ceph-mon[27071]: mgrmap e26: smithi069.wpbewu(active, since 11s), standbys: smithi019.ixuvdm 2024-03-26T01:55:40.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:39 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "mgr metadata", "who": "smithi019.ixuvdm", "id": "smithi019.ixuvdm"}]: dispatch 2024-03-26T01:55:40.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:39 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:40.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:39 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:40.949 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:40 smithi019 ceph-mon[24816]: pgmap v9: 97 pgs: 97 active+clean; 332 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 6.9 MiB/s wr, 651 op/s 2024-03-26T01:55:41.080 INFO:teuthology.orchestra.run.smithi019.stdout: 5 81908 13.47 MB/sec execute 14 sec latency 39.674 ms 2024-03-26T01:55:41.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:40 smithi069 ceph-mon[27071]: pgmap v9: 97 pgs: 97 active+clean; 332 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 6.9 MiB/s wr, 651 op/s 2024-03-26T01:55:41.953 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:41 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:41.953 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:41 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:42.081 INFO:teuthology.orchestra.run.smithi019.stdout: 5 82349 12.94 MB/sec execute 15 sec latency 40.916 ms 2024-03-26T01:55:42.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:41 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:42.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:41 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:42.958 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:42 smithi019 ceph-mon[24816]: pgmap v10: 97 pgs: 97 active+clean; 332 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 6.3 MiB/s wr, 596 op/s 2024-03-26T01:55:43.081 INFO:teuthology.orchestra.run.smithi019.stdout: 5 82910 12.95 MB/sec execute 16 sec latency 36.022 ms 2024-03-26T01:55:43.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:42 smithi069 ceph-mon[27071]: pgmap v10: 97 pgs: 97 active+clean; 332 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 6.3 MiB/s wr, 596 op/s 2024-03-26T01:55:44.081 INFO:teuthology.orchestra.run.smithi019.stdout: 5 83452 12.90 MB/sec execute 17 sec latency 36.493 ms 2024-03-26T01:55:44.717 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:44 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:44.718 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:44 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:44.718 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:44 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:55:44.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:44 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:44.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:44 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:44.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:44 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:55:45.081 INFO:teuthology.orchestra.run.smithi019.stdout: 5 83989 12.56 MB/sec execute 18 sec latency 32.847 ms 2024-03-26T01:55:45.724 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:45 smithi019 ceph-mon[24816]: pgmap v11: 97 pgs: 97 active+clean; 332 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 6.3 MiB/s wr, 596 op/s 2024-03-26T01:55:45.724 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:45 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:45.724 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:45 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:45.724 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:45 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:55:45.724 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:45 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:55:45.724 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:45 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:55:45.724 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:45 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:55:45.724 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:45 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:55:45.724 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:45 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:55:45.724 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:45 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:55:45.724 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:45 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:55:45.725 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:45 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:45.725 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:45 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:45.725 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:45 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:45.725 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:45 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:55:45.725 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:45 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:55:45.725 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:45 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:55:45.725 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:45 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:55:45.725 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:45 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:55:45.725 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:45 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:55:45.725 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:45 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:55:45.725 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:45 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:55:45.725 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:45 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:45.725 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:45 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:55:45.726 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:45 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:55:45.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:45 smithi069 ceph-mon[27071]: pgmap v11: 97 pgs: 97 active+clean; 332 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 6.3 MiB/s wr, 596 op/s 2024-03-26T01:55:45.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:45 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:45.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:45 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:45.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:45 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:55:45.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:45 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:55:45.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:45 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:55:45.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:45 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:55:45.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:45 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:55:45.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:45 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:55:45.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:45 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:55:45.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:45 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:55:45.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:45 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:45.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:45 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:45.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:45 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:45.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:45 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:55:45.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:45 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:55:45.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:45 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:55:45.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:45 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:55:45.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:45 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:55:45.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:45 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:55:45.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:45 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:55:45.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:45 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:55:45.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:45 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:45.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:45 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:55:45.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:45 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:55:46.081 INFO:teuthology.orchestra.run.smithi019.stdout: 5 84461 12.91 MB/sec execute 19 sec latency 22.094 ms 2024-03-26T01:55:46.736 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:46 smithi019 ceph-mon[24816]: Adjusting osd_memory_target on smithi069 to 2704M 2024-03-26T01:55:46.736 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:46 smithi019 ceph-mon[24816]: Adjusting osd_memory_target on smithi019 to 1937M 2024-03-26T01:55:46.736 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:46 smithi019 ceph-mon[24816]: Updating smithi019:/etc/ceph/ceph.conf 2024-03-26T01:55:46.736 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:46 smithi019 ceph-mon[24816]: Updating smithi069:/etc/ceph/ceph.conf 2024-03-26T01:55:46.736 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:46 smithi019 ceph-mon[24816]: Updating smithi069:/var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:55:46.737 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:46 smithi019 ceph-mon[24816]: Updating smithi019:/var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:55:46.737 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:46 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:46.737 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:46 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:46.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:46 smithi069 ceph-mon[27071]: Adjusting osd_memory_target on smithi069 to 2704M 2024-03-26T01:55:46.861 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:46 smithi069 ceph-mon[27071]: Adjusting osd_memory_target on smithi019 to 1937M 2024-03-26T01:55:46.861 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:46 smithi069 ceph-mon[27071]: Updating smithi019:/etc/ceph/ceph.conf 2024-03-26T01:55:46.862 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:46 smithi069 ceph-mon[27071]: Updating smithi069:/etc/ceph/ceph.conf 2024-03-26T01:55:46.862 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:46 smithi069 ceph-mon[27071]: Updating smithi069:/var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:55:46.862 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:46 smithi069 ceph-mon[27071]: Updating smithi019:/var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:55:46.862 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:46 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:46.862 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:46 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:47.081 INFO:teuthology.orchestra.run.smithi019.stdout: 5 85182 13.72 MB/sec execute 20 sec latency 32.688 ms 2024-03-26T01:55:47.742 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:47 smithi019 ceph-mon[24816]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:55:47.742 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:47 smithi019 ceph-mon[24816]: Updating smithi019:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:55:47.742 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:47 smithi019 ceph-mon[24816]: Updating smithi069:/var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T01:55:47.743 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:47 smithi019 ceph-mon[24816]: Updating smithi019:/var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T01:55:47.743 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:47 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:47.743 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:47 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:47.743 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:47 smithi019 ceph-mon[24816]: pgmap v12: 97 pgs: 97 active+clean; 322 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 31 KiB/s rd, 8.9 MiB/s wr, 894 op/s 2024-03-26T01:55:47.743 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:47 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:47.743 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:47 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:47.743 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:47 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:47.743 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:47 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:47.743 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:47 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:47.743 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:47 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:47.743 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:47 smithi019 ceph-mon[24816]: Reconfiguring alertmanager.smithi019 (dependencies changed)... 2024-03-26T01:55:47.743 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:47 smithi019 ceph-mon[24816]: Reconfiguring daemon alertmanager.smithi019 on smithi019 2024-03-26T01:55:47.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:47 smithi069 ceph-mon[27071]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:55:47.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:47 smithi069 ceph-mon[27071]: Updating smithi019:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:55:47.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:47 smithi069 ceph-mon[27071]: Updating smithi069:/var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T01:55:47.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:47 smithi069 ceph-mon[27071]: Updating smithi019:/var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T01:55:47.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:47 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:47.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:47 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:47.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:47 smithi069 ceph-mon[27071]: pgmap v12: 97 pgs: 97 active+clean; 322 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 31 KiB/s rd, 8.9 MiB/s wr, 894 op/s 2024-03-26T01:55:47.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:47 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:47.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:47 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:47.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:47 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:47.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:47 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:47.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:47 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:47.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:47 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:55:47.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:47 smithi069 ceph-mon[27071]: Reconfiguring alertmanager.smithi019 (dependencies changed)... 2024-03-26T01:55:47.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:47 smithi069 ceph-mon[27071]: Reconfiguring daemon alertmanager.smithi019 on smithi019 2024-03-26T01:55:48.081 INFO:teuthology.orchestra.run.smithi019.stdout: 5 85648 13.66 MB/sec execute 21 sec latency 34.263 ms 2024-03-26T01:55:49.005 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:48 smithi019 ceph-mon[24816]: pgmap v13: 97 pgs: 97 active+clean; 322 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 5.7 MiB/s wr, 605 op/s 2024-03-26T01:55:49.081 INFO:teuthology.orchestra.run.smithi019.stdout: 5 86115 13.30 MB/sec execute 22 sec latency 38.329 ms 2024-03-26T01:55:49.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:48 smithi069 ceph-mon[27071]: pgmap v13: 97 pgs: 97 active+clean; 322 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 5.7 MiB/s wr, 605 op/s 2024-03-26T01:55:50.083 INFO:teuthology.orchestra.run.smithi019.stdout: 5 86736 13.36 MB/sec execute 23 sec latency 43.083 ms 2024-03-26T01:55:51.086 INFO:teuthology.orchestra.run.smithi019.stdout: 5 87589 13.52 MB/sec execute 24 sec latency 40.270 ms 2024-03-26T01:55:51.780 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:51 smithi019 ceph-mon[24816]: pgmap v14: 97 pgs: 97 active+clean; 335 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.4 MiB/s wr, 924 op/s 2024-03-26T01:55:51.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:51 smithi069 ceph-mon[27071]: pgmap v14: 97 pgs: 97 active+clean; 335 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.4 MiB/s wr, 924 op/s 2024-03-26T01:55:52.086 INFO:teuthology.orchestra.run.smithi019.stdout: 5 88061 13.67 MB/sec execute 25 sec latency 32.361 ms 2024-03-26T01:55:53.086 INFO:teuthology.orchestra.run.smithi019.stdout: 5 88870 14.53 MB/sec execute 26 sec latency 29.648 ms 2024-03-26T01:55:53.675 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:53 smithi019 ceph-mon[24816]: pgmap v15: 97 pgs: 97 active+clean; 335 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 6.3 MiB/s wr, 617 op/s 2024-03-26T01:55:54.086 INFO:teuthology.orchestra.run.smithi019.stdout: 5 89335 14.21 MB/sec execute 27 sec latency 29.736 ms 2024-03-26T01:55:54.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:53 smithi069 ceph-mon[27071]: pgmap v15: 97 pgs: 97 active+clean; 335 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 6.3 MiB/s wr, 617 op/s 2024-03-26T01:55:55.086 INFO:teuthology.orchestra.run.smithi019.stdout: 5 89789 14.01 MB/sec execute 28 sec latency 26.855 ms 2024-03-26T01:55:55.859 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:55 smithi019 ceph-mon[24816]: pgmap v16: 97 pgs: 97 active+clean; 327 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 7.0 MiB/s wr, 645 op/s 2024-03-26T01:55:56.086 INFO:teuthology.orchestra.run.smithi019.stdout: 5 90293 13.89 MB/sec execute 29 sec latency 37.193 ms 2024-03-26T01:55:56.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:55 smithi069 ceph-mon[27071]: pgmap v16: 97 pgs: 97 active+clean; 327 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 7.0 MiB/s wr, 645 op/s 2024-03-26T01:55:57.086 INFO:teuthology.orchestra.run.smithi019.stdout: 5 90775 13.76 MB/sec execute 30 sec latency 46.978 ms 2024-03-26T01:55:57.925 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:57 smithi019 ceph-mon[24816]: pgmap v17: 97 pgs: 97 active+clean; 333 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 975 op/s 2024-03-26T01:55:58.086 INFO:teuthology.orchestra.run.smithi019.stdout: 5 91237 13.66 MB/sec execute 31 sec latency 36.163 ms 2024-03-26T01:55:58.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:57 smithi069 ceph-mon[27071]: pgmap v17: 97 pgs: 97 active+clean; 333 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 975 op/s 2024-03-26T01:55:58.942 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:58 smithi019 ceph-mon[24816]: pgmap v18: 97 pgs: 97 active+clean; 333 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 7.6 MiB/s wr, 678 op/s 2024-03-26T01:55:58.942 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:55:58 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:55:59.086 INFO:teuthology.orchestra.run.smithi019.stdout: 5 91750 13.93 MB/sec execute 32 sec latency 28.109 ms 2024-03-26T01:55:59.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:58 smithi069 ceph-mon[27071]: pgmap v18: 97 pgs: 97 active+clean; 333 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 7.6 MiB/s wr, 678 op/s 2024-03-26T01:55:59.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:55:58 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:56:00.086 INFO:teuthology.orchestra.run.smithi019.stdout: 5 92335 14.15 MB/sec execute 33 sec latency 24.667 ms 2024-03-26T01:56:01.087 INFO:teuthology.orchestra.run.smithi019.stdout: 5 92717 14.06 MB/sec execute 34 sec latency 37.883 ms 2024-03-26T01:56:01.735 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:01 smithi019 ceph-mon[24816]: pgmap v19: 97 pgs: 97 active+clean; 329 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 11 MiB/s wr, 968 op/s 2024-03-26T01:56:01.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:01 smithi069 ceph-mon[27071]: pgmap v19: 97 pgs: 97 active+clean; 329 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 11 MiB/s wr, 968 op/s 2024-03-26T01:56:02.086 INFO:teuthology.orchestra.run.smithi019.stdout: 5 93183 13.79 MB/sec execute 35 sec latency 44.682 ms 2024-03-26T01:56:03.086 INFO:teuthology.orchestra.run.smithi019.stdout: 5 93706 13.83 MB/sec execute 36 sec latency 30.385 ms 2024-03-26T01:56:03.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:03 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:56:03.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:03 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:56:03.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:03 smithi019 ceph-mon[24816]: pgmap v20: 97 pgs: 97 active+clean; 329 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 6.9 MiB/s wr, 649 op/s 2024-03-26T01:56:03.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:03 smithi019 ceph-mon[24816]: Reconfiguring grafana.smithi019 (dependencies changed)... 2024-03-26T01:56:03.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:03 smithi019 ceph-mon[24816]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T01:56:03.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:03 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:56:03.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:03 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:56:03.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:03 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T01:56:03.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:03 smithi019 ceph-mon[24816]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T01:56:03.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:03 smithi019 ceph-mon[24816]: Reconfiguring daemon grafana.smithi019 on smithi019 2024-03-26T01:56:03.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:03 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:56:03.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:03 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:56:03.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:03 smithi069 ceph-mon[27071]: pgmap v20: 97 pgs: 97 active+clean; 329 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 6.9 MiB/s wr, 649 op/s 2024-03-26T01:56:03.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:03 smithi069 ceph-mon[27071]: Reconfiguring grafana.smithi019 (dependencies changed)... 2024-03-26T01:56:03.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:03 smithi069 ceph-mon[27071]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T01:56:03.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:03 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:56:03.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:03 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:56:03.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:03 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T01:56:03.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:03 smithi069 ceph-mon[27071]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T01:56:03.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:03 smithi069 ceph-mon[27071]: Reconfiguring daemon grafana.smithi019 on smithi019 2024-03-26T01:56:04.087 INFO:teuthology.orchestra.run.smithi019.stdout: 5 94343 13.74 MB/sec execute 37 sec latency 31.063 ms 2024-03-26T01:56:05.087 INFO:teuthology.orchestra.run.smithi019.stdout: 5 94945 13.82 MB/sec execute 38 sec latency 33.734 ms 2024-03-26T01:56:05.785 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:05 smithi019 ceph-mon[24816]: pgmap v21: 97 pgs: 97 active+clean; 324 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.3 MiB/s wr, 669 op/s 2024-03-26T01:56:06.086 INFO:teuthology.orchestra.run.smithi019.stdout: 5 95632 14.22 MB/sec execute 39 sec latency 23.320 ms 2024-03-26T01:56:06.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:05 smithi069 ceph-mon[27071]: pgmap v21: 97 pgs: 97 active+clean; 324 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.3 MiB/s wr, 669 op/s 2024-03-26T01:56:07.087 INFO:teuthology.orchestra.run.smithi019.stdout: 5 96117 14.27 MB/sec execute 40 sec latency 29.979 ms 2024-03-26T01:56:07.527 INFO:teuthology.orchestra.run.smithi019.stdout:true 2024-03-26T01:56:07.827 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:07 smithi019 ceph-mon[24816]: pgmap v22: 97 pgs: 97 active+clean; 322 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 9.0 MiB/s wr, 937 op/s 2024-03-26T01:56:07.910 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:07 smithi069 ceph-mon[27071]: pgmap v22: 97 pgs: 97 active+clean; 322 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 9.0 MiB/s wr, 937 op/s 2024-03-26T01:56:08.087 INFO:teuthology.orchestra.run.smithi019.stdout: 5 96448 14.03 MB/sec execute 41 sec latency 52.051 ms 2024-03-26T01:56:08.642 INFO:teuthology.orchestra.run.smithi019.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T01:56:08.642 INFO:teuthology.orchestra.run.smithi019.stdout:alertmanager.smithi019 smithi019 *:9093,9094 starting - - - - 2024-03-26T01:56:08.642 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi019 smithi019 running (9m) 25s ago 9m 6405k - 17.2.0 e1d6a67b021e 713db693d5c1 2024-03-26T01:56:08.642 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi069 smithi069 running (8m) 29s ago 8m 6405k - 17.2.0 e1d6a67b021e a87d27e2b74b 2024-03-26T01:56:08.642 INFO:teuthology.orchestra.run.smithi019.stdout:grafana.smithi019 smithi019 *:3000 running (7m) 25s ago 9m 53.8M - 8.3.5 dad864ee21e9 81e9f452f143 2024-03-26T01:56:08.642 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi019.lmryom smithi019 running (3m) 25s ago 3m 36.2M - 17.2.0 e1d6a67b021e 91a7c2942715 2024-03-26T01:56:08.642 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi069.spgwgo smithi069 running (3m) 29s ago 3m 12.7M - 17.2.0 e1d6a67b021e 94d96338dbe1 2024-03-26T01:56:08.642 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi019.ixuvdm smithi019 *:9283 running (10m) 25s ago 10m 333M - 17.2.0 e1d6a67b021e 1d13a7db0753 2024-03-26T01:56:08.642 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi069.wpbewu smithi069 *:8443,9283 running (50s) 29s ago 8m 486M - 19.0.0-1775-g5e70124b 5f21c9264136 c0a5be7a9f9e 2024-03-26T01:56:08.642 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi019 smithi019 running (10m) 25s ago 10m 50.5M 2048M 17.2.0 e1d6a67b021e b6908b497829 2024-03-26T01:56:08.643 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi069 smithi069 running (8m) 29s ago 8m 42.7M 2048M 17.2.0 e1d6a67b021e 2d51b7b7bb32 2024-03-26T01:56:08.643 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.0.0.smithi019.qbizxo smithi019 *:2049 running (2m) 25s ago 2m 265M - 4.0 e1d6a67b021e cd6628a9075d 2024-03-26T01:56:08.643 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.1.0.smithi069.uawdhc smithi069 *:2049 running (2m) 29s ago 2m 75.3M - 4.0 e1d6a67b021e 6ee3695e7a10 2024-03-26T01:56:08.643 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi019 smithi019 *:9100 running (8m) 25s ago 8m 20.1M - 1.3.1 1dbe0e931976 8094999d4d38 2024-03-26T01:56:08.643 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi069 smithi069 *:9100 running (8m) 29s ago 8m 22.1M - 1.3.1 1dbe0e931976 46e719c0de82 2024-03-26T01:56:08.643 INFO:teuthology.orchestra.run.smithi019.stdout:osd.0 smithi019 running (7m) 25s ago 7m 323M 1937M 17.2.0 e1d6a67b021e c1a797230ab6 2024-03-26T01:56:08.643 INFO:teuthology.orchestra.run.smithi019.stdout:osd.1 smithi019 running (7m) 25s ago 7m 248M 1937M 17.2.0 e1d6a67b021e 31d3489c7188 2024-03-26T01:56:08.643 INFO:teuthology.orchestra.run.smithi019.stdout:osd.2 smithi019 running (6m) 25s ago 6m 195M 1937M 17.2.0 e1d6a67b021e f549115e7685 2024-03-26T01:56:08.643 INFO:teuthology.orchestra.run.smithi019.stdout:osd.3 smithi019 running (6m) 25s ago 6m 263M 1937M 17.2.0 e1d6a67b021e 154af68d03e1 2024-03-26T01:56:08.643 INFO:teuthology.orchestra.run.smithi019.stdout:osd.4 smithi069 running (5m) 29s ago 5m 281M 2704M 17.2.0 e1d6a67b021e 4fd5e853f8bf 2024-03-26T01:56:08.643 INFO:teuthology.orchestra.run.smithi019.stdout:osd.5 smithi069 running (5m) 29s ago 5m 318M 2704M 17.2.0 e1d6a67b021e 932e401d87c9 2024-03-26T01:56:08.643 INFO:teuthology.orchestra.run.smithi019.stdout:osd.6 smithi069 running (5m) 29s ago 5m 272M 2704M 17.2.0 e1d6a67b021e b123f75154c8 2024-03-26T01:56:08.643 INFO:teuthology.orchestra.run.smithi019.stdout:osd.7 smithi069 running (4m) 29s ago 4m 398M 2704M 17.2.0 e1d6a67b021e d175fbc9ff8a 2024-03-26T01:56:08.643 INFO:teuthology.orchestra.run.smithi019.stdout:prometheus.smithi019 smithi019 *:9095 running (8m) 25s ago 8m 65.1M - 2.33.4 514e6a882f6e 33689b2df9a5 2024-03-26T01:56:08.853 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:08 smithi019 ceph-mon[24816]: from='client.24487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:08.920 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:08 smithi069 ceph-mon[27071]: from='client.24487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:09.087 INFO:teuthology.orchestra.run.smithi019.stdout: 5 96813 13.81 MB/sec execute 42 sec latency 33.077 ms 2024-03-26T01:56:09.340 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T01:56:09.340 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": { 2024-03-26T01:56:09.341 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T01:56:09.341 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:56:09.341 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": { 2024-03-26T01:56:09.341 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-26T01:56:09.341 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 1 2024-03-26T01:56:09.341 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:56:09.341 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": { 2024-03-26T01:56:09.341 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T01:56:09.341 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:56:09.341 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": { 2024-03-26T01:56:09.341 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T01:56:09.341 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:56:09.342 INFO:teuthology.orchestra.run.smithi019.stdout: "overall": { 2024-03-26T01:56:09.342 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-26T01:56:09.342 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 1 2024-03-26T01:56:09.342 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-26T01:56:09.342 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T01:56:09.860 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:09 smithi019 ceph-mon[24816]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:09.860 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:09 smithi019 ceph-mon[24816]: pgmap v23: 97 pgs: 97 active+clean; 322 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 5.8 MiB/s wr, 607 op/s 2024-03-26T01:56:09.860 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:09 smithi019 ceph-mon[24816]: from='client.24495 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:09.860 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:09 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/2228245778' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T01:56:09.934 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T01:56:09.934 INFO:teuthology.orchestra.run.smithi019.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2", 2024-03-26T01:56:09.934 INFO:teuthology.orchestra.run.smithi019.stdout: "in_progress": true, 2024-03-26T01:56:09.934 INFO:teuthology.orchestra.run.smithi019.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T01:56:09.934 INFO:teuthology.orchestra.run.smithi019.stdout: "services_complete": [], 2024-03-26T01:56:09.934 INFO:teuthology.orchestra.run.smithi019.stdout: "progress": "1/23 daemons upgraded", 2024-03-26T01:56:09.935 INFO:teuthology.orchestra.run.smithi019.stdout: "message": "", 2024-03-26T01:56:09.935 INFO:teuthology.orchestra.run.smithi019.stdout: "is_paused": false 2024-03-26T01:56:09.935 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T01:56:10.087 INFO:teuthology.orchestra.run.smithi019.stdout: 5 97256 13.80 MB/sec execute 43 sec latency 42.926 ms 2024-03-26T01:56:10.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:09 smithi069 ceph-mon[27071]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:10.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:09 smithi069 ceph-mon[27071]: pgmap v23: 97 pgs: 97 active+clean; 322 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 5.8 MiB/s wr, 607 op/s 2024-03-26T01:56:10.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:09 smithi069 ceph-mon[27071]: from='client.24495 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:10.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:09 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/2228245778' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T01:56:10.644 INFO:teuthology.orchestra.run.smithi019.stdout:HEALTH_OK 2024-03-26T01:56:10.877 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:10 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/3665186553' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T01:56:11.087 INFO:teuthology.orchestra.run.smithi019.stdout: 5 97769 13.69 MB/sec execute 44 sec latency 43.341 ms 2024-03-26T01:56:11.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:10 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/3665186553' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T01:56:11.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:11 smithi069 ceph-mon[27071]: from='client.14728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:11.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:11 smithi069 ceph-mon[27071]: pgmap v24: 97 pgs: 97 active+clean; 331 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 9.1 MiB/s wr, 897 op/s 2024-03-26T01:56:11.890 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:11 smithi019 ceph-mon[24816]: from='client.14728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:11.891 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:11 smithi019 ceph-mon[24816]: pgmap v24: 97 pgs: 97 active+clean; 331 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 9.1 MiB/s wr, 897 op/s 2024-03-26T01:56:12.087 INFO:teuthology.orchestra.run.smithi019.stdout: 5 98257 13.59 MB/sec execute 45 sec latency 51.427 ms 2024-03-26T01:56:13.087 INFO:teuthology.orchestra.run.smithi019.stdout: 5 98684 13.68 MB/sec execute 46 sec latency 27.899 ms 2024-03-26T01:56:13.704 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:13 smithi019 ceph-mon[24816]: pgmap v25: 97 pgs: 97 active+clean; 331 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.6 KiB/s rd, 6.1 MiB/s wr, 606 op/s 2024-03-26T01:56:14.089 INFO:teuthology.orchestra.run.smithi019.stdout: 5 99138 13.80 MB/sec execute 47 sec latency 37.060 ms 2024-03-26T01:56:14.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:13 smithi069 ceph-mon[27071]: pgmap v25: 97 pgs: 97 active+clean; 331 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.6 KiB/s rd, 6.1 MiB/s wr, 606 op/s 2024-03-26T01:56:14.711 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:14 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:56:15.088 INFO:teuthology.orchestra.run.smithi019.stdout: 5 99533 13.86 MB/sec execute 48 sec latency 49.922 ms 2024-03-26T01:56:15.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:14 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:56:15.717 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:15 smithi019 ceph-mon[24816]: pgmap v26: 97 pgs: 97 active+clean; 327 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.6 KiB/s rd, 6.7 MiB/s wr, 629 op/s 2024-03-26T01:56:16.088 INFO:teuthology.orchestra.run.smithi019.stdout: 5 99784 13.65 MB/sec execute 49 sec latency 36.466 ms 2024-03-26T01:56:16.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:15 smithi069 ceph-mon[27071]: pgmap v26: 97 pgs: 97 active+clean; 327 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.6 KiB/s rd, 6.7 MiB/s wr, 629 op/s 2024-03-26T01:56:17.089 INFO:teuthology.orchestra.run.smithi019.stdout: 5 100050 13.43 MB/sec execute 50 sec latency 40.182 ms 2024-03-26T01:56:17.760 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:17 smithi019 ceph-mon[24816]: pgmap v27: 97 pgs: 97 active+clean; 331 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8.6 MiB/s wr, 852 op/s 2024-03-26T01:56:18.089 INFO:teuthology.orchestra.run.smithi019.stdout: 5 100431 13.33 MB/sec execute 51 sec latency 55.393 ms 2024-03-26T01:56:18.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:17 smithi069 ceph-mon[27071]: pgmap v27: 97 pgs: 97 active+clean; 331 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8.6 MiB/s wr, 852 op/s 2024-03-26T01:56:19.022 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:18 smithi019 ceph-mon[24816]: pgmap v28: 97 pgs: 97 active+clean; 331 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.3 MiB/s wr, 555 op/s 2024-03-26T01:56:19.089 INFO:teuthology.orchestra.run.smithi019.stdout: 5 100824 13.25 MB/sec execute 52 sec latency 51.015 ms 2024-03-26T01:56:19.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:18 smithi069 ceph-mon[27071]: pgmap v28: 97 pgs: 97 active+clean; 331 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.3 MiB/s wr, 555 op/s 2024-03-26T01:56:20.089 INFO:teuthology.orchestra.run.smithi019.stdout: 5 101408 13.20 MB/sec execute 53 sec latency 38.272 ms 2024-03-26T01:56:21.089 INFO:teuthology.orchestra.run.smithi019.stdout: 5 101889 13.19 MB/sec execute 54 sec latency 26.690 ms 2024-03-26T01:56:21.823 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:21 smithi019 ceph-mon[24816]: pgmap v29: 97 pgs: 97 active+clean; 326 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.2 MiB/s wr, 785 op/s 2024-03-26T01:56:21.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:21 smithi069 ceph-mon[27071]: pgmap v29: 97 pgs: 97 active+clean; 326 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.2 MiB/s wr, 785 op/s 2024-03-26T01:56:22.089 INFO:teuthology.orchestra.run.smithi019.stdout: 5 102284 13.20 MB/sec execute 55 sec latency 30.691 ms 2024-03-26T01:56:23.089 INFO:teuthology.orchestra.run.smithi019.stdout: 5 102734 13.37 MB/sec execute 56 sec latency 32.031 ms 2024-03-26T01:56:23.862 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:23 smithi019 ceph-mon[24816]: pgmap v30: 97 pgs: 97 active+clean; 326 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 4.9 MiB/s wr, 496 op/s 2024-03-26T01:56:24.089 INFO:teuthology.orchestra.run.smithi019.stdout: 5 103086 13.31 MB/sec execute 57 sec latency 30.815 ms 2024-03-26T01:56:24.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:23 smithi069 ceph-mon[27071]: pgmap v30: 97 pgs: 97 active+clean; 326 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 4.9 MiB/s wr, 496 op/s 2024-03-26T01:56:25.089 INFO:teuthology.orchestra.run.smithi019.stdout: 5 103402 13.19 MB/sec execute 58 sec latency 37.273 ms 2024-03-26T01:56:25.892 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:25 smithi019 ceph-mon[24816]: pgmap v31: 97 pgs: 97 active+clean; 331 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.7 MiB/s wr, 572 op/s 2024-03-26T01:56:25.892 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:25 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:56:25.892 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:25 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:56:26.089 INFO:teuthology.orchestra.run.smithi019.stdout: 5 103716 13.03 MB/sec execute 59 sec latency 82.690 ms 2024-03-26T01:56:26.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:25 smithi069 ceph-mon[27071]: pgmap v31: 97 pgs: 97 active+clean; 331 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.7 MiB/s wr, 572 op/s 2024-03-26T01:56:26.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:25 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:56:26.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:25 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:56:26.910 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:26 smithi019 ceph-mon[24816]: Reconfiguring node-exporter.smithi019 (dependencies changed)... 2024-03-26T01:56:26.910 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:26 smithi019 ceph-mon[24816]: Reconfiguring daemon node-exporter.smithi019 on smithi019 2024-03-26T01:56:27.089 INFO:teuthology.orchestra.run.smithi019.stdout: 5 104084 12.98 MB/sec execute 60 sec latency 47.092 ms 2024-03-26T01:56:27.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:26 smithi069 ceph-mon[27071]: Reconfiguring node-exporter.smithi019 (dependencies changed)... 2024-03-26T01:56:27.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:26 smithi069 ceph-mon[27071]: Reconfiguring daemon node-exporter.smithi019 on smithi019 2024-03-26T01:56:27.921 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:27 smithi019 ceph-mon[24816]: pgmap v32: 97 pgs: 97 active+clean; 331 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 717 op/s 2024-03-26T01:56:28.089 INFO:teuthology.orchestra.run.smithi019.stdout: 5 104467 12.87 MB/sec execute 61 sec latency 63.491 ms 2024-03-26T01:56:28.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:27 smithi069 ceph-mon[27071]: pgmap v32: 97 pgs: 97 active+clean; 331 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 717 op/s 2024-03-26T01:56:29.089 INFO:teuthology.orchestra.run.smithi019.stdout: 5 104968 12.83 MB/sec execute 62 sec latency 78.942 ms 2024-03-26T01:56:29.183 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:28 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:56:29.183 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:28 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:56:29.183 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:28 smithi019 ceph-mon[24816]: Reconfiguring prometheus.smithi019 (dependencies changed)... 2024-03-26T01:56:29.183 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:28 smithi019 ceph-mon[24816]: Reconfiguring daemon prometheus.smithi019 on smithi019 2024-03-26T01:56:29.183 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:28 smithi019 ceph-mon[24816]: pgmap v33: 97 pgs: 97 active+clean; 331 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.8 MiB/s wr, 473 op/s 2024-03-26T01:56:29.183 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:28 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:56:29.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:28 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:56:29.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:28 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:56:29.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:28 smithi069 ceph-mon[27071]: Reconfiguring prometheus.smithi019 (dependencies changed)... 2024-03-26T01:56:29.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:28 smithi069 ceph-mon[27071]: Reconfiguring daemon prometheus.smithi019 on smithi019 2024-03-26T01:56:29.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:28 smithi069 ceph-mon[27071]: pgmap v33: 97 pgs: 97 active+clean; 331 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.8 MiB/s wr, 473 op/s 2024-03-26T01:56:29.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:28 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:56:30.090 INFO:teuthology.orchestra.run.smithi019.stdout: 5 105524 12.87 MB/sec execute 63 sec latency 33.052 ms 2024-03-26T01:56:31.090 INFO:teuthology.orchestra.run.smithi019.stdout: 5 105961 12.91 MB/sec execute 64 sec latency 33.162 ms 2024-03-26T01:56:31.702 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:31 smithi019 ceph-mon[24816]: pgmap v34: 97 pgs: 97 active+clean; 320 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 6.8 MiB/s wr, 723 op/s 2024-03-26T01:56:31.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:31 smithi069 ceph-mon[27071]: pgmap v34: 97 pgs: 97 active+clean; 320 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 6.8 MiB/s wr, 723 op/s 2024-03-26T01:56:32.090 INFO:teuthology.orchestra.run.smithi019.stdout: 5 106385 13.02 MB/sec execute 65 sec latency 52.973 ms 2024-03-26T01:56:33.090 INFO:teuthology.orchestra.run.smithi019.stdout: 5 106774 13.00 MB/sec execute 66 sec latency 61.634 ms 2024-03-26T01:56:33.729 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:33 smithi019 ceph-mon[24816]: pgmap v35: 97 pgs: 97 active+clean; 320 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.9 MiB/s wr, 493 op/s 2024-03-26T01:56:34.090 INFO:teuthology.orchestra.run.smithi019.stdout: 5 107038 12.86 MB/sec execute 67 sec latency 79.346 ms 2024-03-26T01:56:34.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:33 smithi069 ceph-mon[27071]: pgmap v35: 97 pgs: 97 active+clean; 320 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.9 MiB/s wr, 493 op/s 2024-03-26T01:56:35.090 INFO:teuthology.orchestra.run.smithi019.stdout: 5 107350 12.73 MB/sec execute 68 sec latency 46.842 ms 2024-03-26T01:56:35.747 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:35 smithi019 ceph-mon[24816]: pgmap v36: 97 pgs: 97 active+clean; 318 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.4 MiB/s wr, 507 op/s 2024-03-26T01:56:36.090 INFO:teuthology.orchestra.run.smithi019.stdout: 5 107673 12.67 MB/sec execute 69 sec latency 75.448 ms 2024-03-26T01:56:36.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:35 smithi069 ceph-mon[27071]: pgmap v36: 97 pgs: 97 active+clean; 318 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.4 MiB/s wr, 507 op/s 2024-03-26T01:56:37.090 INFO:teuthology.orchestra.run.smithi019.stdout: 5 108136 12.61 MB/sec execute 70 sec latency 45.863 ms 2024-03-26T01:56:37.780 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:37 smithi019 ceph-mon[24816]: pgmap v37: 97 pgs: 97 active+clean; 331 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 6.9 MiB/s wr, 650 op/s 2024-03-26T01:56:38.090 INFO:teuthology.orchestra.run.smithi019.stdout: 5 108626 12.57 MB/sec execute 71 sec latency 108.330 ms 2024-03-26T01:56:38.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:37 smithi069 ceph-mon[27071]: pgmap v37: 97 pgs: 97 active+clean; 331 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 6.9 MiB/s wr, 650 op/s 2024-03-26T01:56:38.864 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:38 smithi019 ceph-mon[24816]: pgmap v38: 97 pgs: 97 active+clean; 331 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.9 MiB/s wr, 483 op/s 2024-03-26T01:56:39.090 INFO:teuthology.orchestra.run.smithi019.stdout: 5 109172 12.63 MB/sec execute 72 sec latency 68.407 ms 2024-03-26T01:56:39.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:38 smithi069 ceph-mon[27071]: pgmap v38: 97 pgs: 97 active+clean; 331 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.9 MiB/s wr, 483 op/s 2024-03-26T01:56:40.091 INFO:teuthology.orchestra.run.smithi019.stdout: 5 109895 12.86 MB/sec execute 73 sec latency 29.707 ms 2024-03-26T01:56:41.091 INFO:teuthology.orchestra.run.smithi019.stdout: 5 110543 12.93 MB/sec execute 74 sec latency 32.828 ms 2024-03-26T01:56:41.240 INFO:teuthology.orchestra.run.smithi019.stdout:true 2024-03-26T01:56:41.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:41 smithi069 ceph-mon[27071]: pgmap v39: 97 pgs: 97 active+clean; 325 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.5 MiB/s wr, 761 op/s 2024-03-26T01:56:41.879 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:41 smithi019 ceph-mon[24816]: pgmap v39: 97 pgs: 97 active+clean; 325 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.5 MiB/s wr, 761 op/s 2024-03-26T01:56:42.090 INFO:teuthology.orchestra.run.smithi019.stdout: 5 110979 12.86 MB/sec execute 75 sec latency 32.673 ms 2024-03-26T01:56:42.365 INFO:teuthology.orchestra.run.smithi019.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T01:56:42.365 INFO:teuthology.orchestra.run.smithi019.stdout:alertmanager.smithi019 smithi019 *:9093,9094 starting - - - - 2024-03-26T01:56:42.366 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi019 smithi019 running (9m) 58s ago 9m 6405k - 17.2.0 e1d6a67b021e 713db693d5c1 2024-03-26T01:56:42.366 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi069 smithi069 running (9m) 62s ago 9m 6405k - 17.2.0 e1d6a67b021e a87d27e2b74b 2024-03-26T01:56:42.366 INFO:teuthology.orchestra.run.smithi019.stdout:grafana.smithi019 smithi019 *:3000 starting - - - - 2024-03-26T01:56:42.366 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi019.lmryom smithi019 running (3m) 58s ago 3m 36.2M - 17.2.0 e1d6a67b021e 91a7c2942715 2024-03-26T01:56:42.366 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi069.spgwgo smithi069 running (3m) 62s ago 3m 12.7M - 17.2.0 e1d6a67b021e 94d96338dbe1 2024-03-26T01:56:42.366 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi019.ixuvdm smithi019 *:9283 running (11m) 58s ago 11m 333M - 17.2.0 e1d6a67b021e 1d13a7db0753 2024-03-26T01:56:42.366 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi069.wpbewu smithi069 *:8443,9283 running (84s) 62s ago 9m 486M - 19.0.0-1775-g5e70124b 5f21c9264136 c0a5be7a9f9e 2024-03-26T01:56:42.366 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi019 smithi019 running (11m) 58s ago 11m 50.5M 2048M 17.2.0 e1d6a67b021e b6908b497829 2024-03-26T01:56:42.366 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi069 smithi069 running (9m) 62s ago 9m 42.7M 2048M 17.2.0 e1d6a67b021e 2d51b7b7bb32 2024-03-26T01:56:42.366 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.0.0.smithi019.qbizxo smithi019 *:2049 running (3m) 58s ago 3m 265M - 4.0 e1d6a67b021e cd6628a9075d 2024-03-26T01:56:42.366 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.1.0.smithi069.uawdhc smithi069 *:2049 running (3m) 62s ago 3m 75.3M - 4.0 e1d6a67b021e 6ee3695e7a10 2024-03-26T01:56:42.366 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi019 smithi019 *:9100 starting - - - - 2024-03-26T01:56:42.366 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi069 smithi069 *:9100 running (8m) 62s ago 8m 22.1M - 1.3.1 1dbe0e931976 46e719c0de82 2024-03-26T01:56:42.367 INFO:teuthology.orchestra.run.smithi019.stdout:osd.0 smithi019 running (8m) 58s ago 8m 323M 1937M 17.2.0 e1d6a67b021e c1a797230ab6 2024-03-26T01:56:42.367 INFO:teuthology.orchestra.run.smithi019.stdout:osd.1 smithi019 running (7m) 58s ago 7m 248M 1937M 17.2.0 e1d6a67b021e 31d3489c7188 2024-03-26T01:56:42.367 INFO:teuthology.orchestra.run.smithi019.stdout:osd.2 smithi019 running (7m) 58s ago 7m 195M 1937M 17.2.0 e1d6a67b021e f549115e7685 2024-03-26T01:56:42.367 INFO:teuthology.orchestra.run.smithi019.stdout:osd.3 smithi019 running (6m) 58s ago 6m 263M 1937M 17.2.0 e1d6a67b021e 154af68d03e1 2024-03-26T01:56:42.367 INFO:teuthology.orchestra.run.smithi019.stdout:osd.4 smithi069 running (6m) 62s ago 6m 281M 2704M 17.2.0 e1d6a67b021e 4fd5e853f8bf 2024-03-26T01:56:42.367 INFO:teuthology.orchestra.run.smithi019.stdout:osd.5 smithi069 running (6m) 62s ago 6m 318M 2704M 17.2.0 e1d6a67b021e 932e401d87c9 2024-03-26T01:56:42.367 INFO:teuthology.orchestra.run.smithi019.stdout:osd.6 smithi069 running (5m) 62s ago 5m 272M 2704M 17.2.0 e1d6a67b021e b123f75154c8 2024-03-26T01:56:42.367 INFO:teuthology.orchestra.run.smithi019.stdout:osd.7 smithi069 running (5m) 62s ago 5m 398M 2704M 17.2.0 e1d6a67b021e d175fbc9ff8a 2024-03-26T01:56:42.367 INFO:teuthology.orchestra.run.smithi019.stdout:prometheus.smithi019 smithi019 *:9095 running (8m) 58s ago 8m 65.1M - 2.33.4 514e6a882f6e 33689b2df9a5 2024-03-26T01:56:42.885 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:42 smithi019 ceph-mon[24816]: from='client.24509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:43.091 INFO:teuthology.orchestra.run.smithi019.stdout: 5 111434 12.80 MB/sec execute 76 sec latency 38.386 ms 2024-03-26T01:56:43.093 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T01:56:43.093 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": { 2024-03-26T01:56:43.093 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T01:56:43.093 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:56:43.093 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": { 2024-03-26T01:56:43.094 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-26T01:56:43.094 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 1 2024-03-26T01:56:43.094 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:56:43.094 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": { 2024-03-26T01:56:43.094 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T01:56:43.094 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:56:43.094 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": { 2024-03-26T01:56:43.094 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T01:56:43.094 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:56:43.094 INFO:teuthology.orchestra.run.smithi019.stdout: "overall": { 2024-03-26T01:56:43.094 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-26T01:56:43.094 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 1 2024-03-26T01:56:43.094 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-26T01:56:43.095 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T01:56:43.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:42 smithi069 ceph-mon[27071]: from='client.24509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:43.554 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T01:56:43.554 INFO:teuthology.orchestra.run.smithi019.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2", 2024-03-26T01:56:43.555 INFO:teuthology.orchestra.run.smithi019.stdout: "in_progress": true, 2024-03-26T01:56:43.555 INFO:teuthology.orchestra.run.smithi019.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T01:56:43.555 INFO:teuthology.orchestra.run.smithi019.stdout: "services_complete": [], 2024-03-26T01:56:43.555 INFO:teuthology.orchestra.run.smithi019.stdout: "progress": "1/23 daemons upgraded", 2024-03-26T01:56:43.555 INFO:teuthology.orchestra.run.smithi019.stdout: "message": "", 2024-03-26T01:56:43.555 INFO:teuthology.orchestra.run.smithi019.stdout: "is_paused": false 2024-03-26T01:56:43.555 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T01:56:43.775 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:43 smithi019 ceph-mon[24816]: from='client.24513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:43.775 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:43 smithi019 ceph-mon[24816]: from='client.24517 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:43.775 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:43 smithi019 ceph-mon[24816]: pgmap v40: 97 pgs: 97 active+clean; 325 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.5 MiB/s wr, 511 op/s 2024-03-26T01:56:43.775 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:43 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/377189446' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T01:56:44.076 INFO:teuthology.orchestra.run.smithi019.stdout:HEALTH_OK 2024-03-26T01:56:44.090 INFO:teuthology.orchestra.run.smithi019.stdout: 5 111816 12.76 MB/sec execute 77 sec latency 69.925 ms 2024-03-26T01:56:44.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:43 smithi069 ceph-mon[27071]: from='client.24513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:44.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:43 smithi069 ceph-mon[27071]: from='client.24517 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:44.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:43 smithi069 ceph-mon[27071]: pgmap v40: 97 pgs: 97 active+clean; 325 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.5 MiB/s wr, 511 op/s 2024-03-26T01:56:44.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:43 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/377189446' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T01:56:44.796 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:44 smithi019 ceph-mon[24816]: from='client.14750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:44.796 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:44 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:56:44.796 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:44 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/1895742558' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T01:56:45.092 INFO:teuthology.orchestra.run.smithi019.stdout: 5 112176 12.67 MB/sec execute 78 sec latency 82.825 ms 2024-03-26T01:56:45.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:44 smithi069 ceph-mon[27071]: from='client.14750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:45.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:44 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:56:45.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:44 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/1895742558' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T01:56:45.814 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:45 smithi019 ceph-mon[24816]: pgmap v41: 97 pgs: 97 active+clean; 320 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 6.3 MiB/s wr, 554 op/s 2024-03-26T01:56:46.092 INFO:teuthology.orchestra.run.smithi019.stdout: 5 112600 12.68 MB/sec execute 79 sec latency 34.357 ms 2024-03-26T01:56:46.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:45 smithi069 ceph-mon[27071]: pgmap v41: 97 pgs: 97 active+clean; 320 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 6.3 MiB/s wr, 554 op/s 2024-03-26T01:56:47.092 INFO:teuthology.orchestra.run.smithi019.stdout: 5 112961 12.69 MB/sec execute 80 sec latency 42.921 ms 2024-03-26T01:56:47.875 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:47 smithi019 ceph-mon[24816]: pgmap v42: 97 pgs: 97 active+clean; 323 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.9 MiB/s wr, 800 op/s 2024-03-26T01:56:48.094 INFO:teuthology.orchestra.run.smithi019.stdout: 5 113398 12.77 MB/sec execute 81 sec latency 28.090 ms 2024-03-26T01:56:48.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:47 smithi069 ceph-mon[27071]: pgmap v42: 97 pgs: 97 active+clean; 323 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.9 MiB/s wr, 800 op/s 2024-03-26T01:56:48.896 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:48 smithi019 ceph-mon[24816]: pgmap v43: 97 pgs: 97 active+clean; 323 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.5 MiB/s wr, 580 op/s 2024-03-26T01:56:49.092 INFO:teuthology.orchestra.run.smithi019.stdout: 5 113774 12.79 MB/sec execute 82 sec latency 32.670 ms 2024-03-26T01:56:49.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:48 smithi069 ceph-mon[27071]: pgmap v43: 97 pgs: 97 active+clean; 323 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.5 MiB/s wr, 580 op/s 2024-03-26T01:56:50.093 INFO:teuthology.orchestra.run.smithi019.stdout: 5 114087 12.68 MB/sec execute 83 sec latency 39.206 ms 2024-03-26T01:56:51.092 INFO:teuthology.orchestra.run.smithi019.stdout: 5 114410 12.58 MB/sec execute 84 sec latency 33.872 ms 2024-03-26T01:56:51.687 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:51 smithi019 ceph-mon[24816]: pgmap v44: 97 pgs: 97 active+clean; 333 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.2 MiB/s wr, 802 op/s 2024-03-26T01:56:51.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:51 smithi069 ceph-mon[27071]: pgmap v44: 97 pgs: 97 active+clean; 333 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.2 MiB/s wr, 802 op/s 2024-03-26T01:56:52.092 INFO:teuthology.orchestra.run.smithi019.stdout: 5 114779 12.54 MB/sec execute 85 sec latency 41.129 ms 2024-03-26T01:56:53.093 INFO:teuthology.orchestra.run.smithi019.stdout: 5 115266 12.52 MB/sec execute 86 sec latency 56.847 ms 2024-03-26T01:56:53.717 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:53 smithi019 ceph-mon[24816]: pgmap v45: 97 pgs: 97 active+clean; 333 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.6 MiB/s wr, 524 op/s 2024-03-26T01:56:54.093 INFO:teuthology.orchestra.run.smithi019.stdout: 5 115623 12.45 MB/sec execute 87 sec latency 48.662 ms 2024-03-26T01:56:54.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:53 smithi069 ceph-mon[27071]: pgmap v45: 97 pgs: 97 active+clean; 333 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.6 MiB/s wr, 524 op/s 2024-03-26T01:56:55.093 INFO:teuthology.orchestra.run.smithi019.stdout: 5 116085 12.45 MB/sec execute 88 sec latency 39.968 ms 2024-03-26T01:56:55.257 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:55 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:56:55.257 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:55 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:56:55.257 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:55 smithi019 ceph-mon[24816]: Reconfiguring node-exporter.smithi069 (dependencies changed)... 2024-03-26T01:56:55.257 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:55 smithi019 ceph-mon[24816]: Reconfiguring daemon node-exporter.smithi069 on smithi069 2024-03-26T01:56:55.257 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:55 smithi019 ceph-mon[24816]: pgmap v46: 97 pgs: 97 active+clean; 329 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.4 KiB/s rd, 6.1 MiB/s wr, 555 op/s 2024-03-26T01:56:55.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:55 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:56:55.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:55 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:56:55.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:55 smithi069 ceph-mon[27071]: Reconfiguring node-exporter.smithi069 (dependencies changed)... 2024-03-26T01:56:55.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:55 smithi069 ceph-mon[27071]: Reconfiguring daemon node-exporter.smithi069 on smithi069 2024-03-26T01:56:55.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:55 smithi069 ceph-mon[27071]: pgmap v46: 97 pgs: 97 active+clean; 329 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.4 KiB/s rd, 6.1 MiB/s wr, 555 op/s 2024-03-26T01:56:56.093 INFO:teuthology.orchestra.run.smithi019.stdout: 5 116506 12.47 MB/sec execute 89 sec latency 37.249 ms 2024-03-26T01:56:57.093 INFO:teuthology.orchestra.run.smithi019.stdout: 5 116947 12.55 MB/sec execute 90 sec latency 48.640 ms 2024-03-26T01:56:57.292 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:57 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:56:57.292 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:57 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:56:57.292 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:57 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T01:56:57.293 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:57 smithi019 ceph-mon[24816]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T01:56:57.293 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:57 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi019.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T01:56:57.293 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:57 smithi019 ceph-mon[24816]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi019.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T01:56:57.293 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:57 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:56:57.293 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:57 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T01:56:57.293 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:57 smithi019 ceph-mon[24816]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T01:56:57.293 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:57 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi019.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T01:56:57.293 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:57 smithi019 ceph-mon[24816]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi019.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T01:56:57.293 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:57 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:56:57.293 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:57 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T01:56:57.294 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:57 smithi019 ceph-mon[24816]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T01:56:57.294 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:57 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi019.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T01:56:57.294 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:57 smithi019 ceph-mon[24816]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi019.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T01:56:57.294 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:57 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:56:57.294 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:57 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:56:57.294 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:57 smithi019 ceph-mon[24816]: Upgrade: Need to upgrade myself (mgr.smithi069.wpbewu) 2024-03-26T01:56:57.294 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:57 smithi019 ceph-mon[24816]: Upgrade: Need to upgrade myself (mgr.smithi069.wpbewu) 2024-03-26T01:56:57.294 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:57 smithi019 ceph-mon[24816]: pgmap v47: 97 pgs: 97 active+clean; 325 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 6.7 MiB/s wr, 725 op/s 2024-03-26T01:56:57.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:57 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:56:57.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:57 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:56:57.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:57 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T01:56:57.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:57 smithi069 ceph-mon[27071]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T01:56:57.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:57 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi019.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T01:56:57.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:57 smithi069 ceph-mon[27071]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi019.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T01:56:57.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:57 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:56:57.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:57 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T01:56:57.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:57 smithi069 ceph-mon[27071]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T01:56:57.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:57 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi019.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T01:56:57.611 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:57 smithi069 ceph-mon[27071]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi019.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T01:56:57.611 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:57 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:56:57.611 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:57 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T01:56:57.611 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:57 smithi069 ceph-mon[27071]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T01:56:57.611 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:57 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi019.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T01:56:57.611 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:57 smithi069 ceph-mon[27071]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi019.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T01:56:57.612 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:57 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:56:57.612 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:57 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:56:57.612 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:57 smithi069 ceph-mon[27071]: Upgrade: Need to upgrade myself (mgr.smithi069.wpbewu) 2024-03-26T01:56:57.612 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:57 smithi069 ceph-mon[27071]: Upgrade: Need to upgrade myself (mgr.smithi069.wpbewu) 2024-03-26T01:56:57.612 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:57 smithi069 ceph-mon[27071]: pgmap v47: 97 pgs: 97 active+clean; 325 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 6.7 MiB/s wr, 725 op/s 2024-03-26T01:56:58.093 INFO:teuthology.orchestra.run.smithi019.stdout: 5 117530 12.60 MB/sec execute 91 sec latency 30.077 ms 2024-03-26T01:56:59.054 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:58 smithi019 ceph-mon[24816]: pgmap v48: 97 pgs: 97 active+clean; 325 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.7 MiB/s wr, 465 op/s 2024-03-26T01:56:59.054 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:56:58 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:56:59.093 INFO:teuthology.orchestra.run.smithi019.stdout: 5 118201 12.60 MB/sec execute 92 sec latency 28.760 ms 2024-03-26T01:56:59.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:58 smithi069 ceph-mon[27071]: pgmap v48: 97 pgs: 97 active+clean; 325 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.7 MiB/s wr, 465 op/s 2024-03-26T01:56:59.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:56:58 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:57:00.093 INFO:teuthology.orchestra.run.smithi019.stdout: 5 118747 12.59 MB/sec execute 93 sec latency 31.735 ms 2024-03-26T01:57:00.328 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:00 smithi019 ceph-mon[24816]: Upgrade: Updating mgr.smithi019.ixuvdm 2024-03-26T01:57:00.328 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:00 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:57:00.328 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:00 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi019.ixuvdm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T01:57:00.328 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:00 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi019.ixuvdm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T01:57:00.328 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:00 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T01:57:00.328 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:00 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:00.328 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:00 smithi019 ceph-mon[24816]: Deploying daemon mgr.smithi019.ixuvdm on smithi019 2024-03-26T01:57:00.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:00 smithi069 ceph-mon[27071]: Upgrade: Updating mgr.smithi019.ixuvdm 2024-03-26T01:57:00.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:00 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:57:00.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:00 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi019.ixuvdm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T01:57:00.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:00 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi019.ixuvdm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T01:57:00.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:00 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T01:57:00.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:00 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:00.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:00 smithi069 ceph-mon[27071]: Deploying daemon mgr.smithi019.ixuvdm on smithi019 2024-03-26T01:57:01.093 INFO:teuthology.orchestra.run.smithi019.stdout: 5 119232 12.54 MB/sec execute 94 sec latency 72.827 ms 2024-03-26T01:57:01.345 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:01 smithi019 ceph-mon[24816]: pgmap v49: 97 pgs: 97 active+clean; 340 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 9.1 KiB/s rd, 8.2 MiB/s wr, 773 op/s 2024-03-26T01:57:01.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:01 smithi069 ceph-mon[27071]: pgmap v49: 97 pgs: 97 active+clean; 340 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 9.1 KiB/s rd, 8.2 MiB/s wr, 773 op/s 2024-03-26T01:57:02.093 INFO:teuthology.orchestra.run.smithi019.stdout: 5 119789 12.58 MB/sec execute 95 sec latency 88.128 ms 2024-03-26T01:57:02.355 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:02 smithi019 ceph-mon[24816]: mgrmap e27: smithi069.wpbewu(active, since 93s), standbys: smithi019.ixuvdm 2024-03-26T01:57:02.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:02 smithi069 ceph-mon[27071]: mgrmap e27: smithi069.wpbewu(active, since 93s), standbys: smithi019.ixuvdm 2024-03-26T01:57:03.093 INFO:teuthology.orchestra.run.smithi019.stdout: 5 120341 12.69 MB/sec execute 96 sec latency 34.489 ms 2024-03-26T01:57:03.360 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:03 smithi019 ceph-mon[24816]: pgmap v50: 97 pgs: 97 active+clean; 340 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.4 MiB/s wr, 551 op/s 2024-03-26T01:57:03.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:03 smithi069 ceph-mon[27071]: pgmap v50: 97 pgs: 97 active+clean; 340 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.9 KiB/s rd, 5.4 MiB/s wr, 551 op/s 2024-03-26T01:57:04.093 INFO:teuthology.orchestra.run.smithi019.stdout: 5 120728 12.72 MB/sec execute 97 sec latency 45.446 ms 2024-03-26T01:57:05.093 INFO:teuthology.orchestra.run.smithi019.stdout: 5 121324 12.71 MB/sec execute 98 sec latency 33.999 ms 2024-03-26T01:57:05.871 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:05 smithi019 ceph-mon[24816]: pgmap v51: 97 pgs: 97 active+clean; 339 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.9 KiB/s rd, 6.5 MiB/s wr, 599 op/s 2024-03-26T01:57:06.093 INFO:teuthology.orchestra.run.smithi019.stdout: 5 122024 12.71 MB/sec execute 99 sec latency 61.875 ms 2024-03-26T01:57:06.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:05 smithi069 ceph-mon[27071]: pgmap v51: 97 pgs: 97 active+clean; 339 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.9 KiB/s rd, 6.5 MiB/s wr, 599 op/s 2024-03-26T01:57:07.093 INFO:teuthology.orchestra.run.smithi019.stdout: 5 122735 12.74 MB/sec execute 100 sec latency 28.086 ms 2024-03-26T01:57:07.729 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:07 smithi019 ceph-mon[24816]: pgmap v52: 97 pgs: 97 active+clean; 334 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.2 MiB/s wr, 815 op/s 2024-03-26T01:57:08.093 INFO:teuthology.orchestra.run.smithi019.stdout: 5 123287 12.76 MB/sec execute 101 sec latency 32.242 ms 2024-03-26T01:57:08.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:07 smithi069 ceph-mon[27071]: pgmap v52: 97 pgs: 97 active+clean; 334 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.2 MiB/s wr, 815 op/s 2024-03-26T01:57:09.093 INFO:teuthology.orchestra.run.smithi019.stdout: 5 123699 12.83 MB/sec execute 102 sec latency 41.574 ms 2024-03-26T01:57:09.274 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:09 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:57:09.274 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:09 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:57:09.274 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:09 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:09.274 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:09 smithi019 ceph-mon[24816]: pgmap v53: 97 pgs: 97 active+clean; 334 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 6.7 MiB/s wr, 602 op/s 2024-03-26T01:57:09.274 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:09 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:57:09.275 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:09 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:57:09.483 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:09 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:57:09.483 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:09 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:57:09.483 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:09 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:09.483 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:09 smithi069 ceph-mon[27071]: pgmap v53: 97 pgs: 97 active+clean; 334 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 6.7 MiB/s wr, 602 op/s 2024-03-26T01:57:09.483 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:09 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:57:09.484 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:09 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:57:10.093 INFO:teuthology.orchestra.run.smithi019.stdout: 5 124044 12.83 MB/sec execute 103 sec latency 41.485 ms 2024-03-26T01:57:11.094 INFO:teuthology.orchestra.run.smithi019.stdout: 5 124530 12.85 MB/sec execute 104 sec latency 23.384 ms 2024-03-26T01:57:11.795 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:11 smithi019 ceph-mon[24816]: pgmap v54: 97 pgs: 97 active+clean; 333 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 941 op/s 2024-03-26T01:57:12.094 INFO:teuthology.orchestra.run.smithi019.stdout: 5 125006 12.79 MB/sec execute 105 sec latency 46.226 ms 2024-03-26T01:57:12.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:11 smithi069 ceph-mon[27071]: pgmap v54: 97 pgs: 97 active+clean; 333 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 941 op/s 2024-03-26T01:57:13.095 INFO:teuthology.orchestra.run.smithi019.stdout: 5 125405 12.75 MB/sec execute 106 sec latency 35.421 ms 2024-03-26T01:57:13.849 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:13 smithi019 ceph-mon[24816]: pgmap v55: 97 pgs: 97 active+clean; 333 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.5 MiB/s wr, 632 op/s 2024-03-26T01:57:13.849 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:13 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:57:13.849 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:13 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:57:14.001 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:13 smithi069 ceph-mon[27071]: pgmap v55: 97 pgs: 97 active+clean; 333 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.5 MiB/s wr, 632 op/s 2024-03-26T01:57:14.001 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:13 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:57:14.001 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:13 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:57:14.094 INFO:teuthology.orchestra.run.smithi019.stdout: 5 125780 12.68 MB/sec execute 107 sec latency 44.484 ms 2024-03-26T01:57:14.617 INFO:teuthology.orchestra.run.smithi019.stdout:true 2024-03-26T01:57:14.872 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:14 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:57:15.095 INFO:teuthology.orchestra.run.smithi019.stdout: 5 126165 12.68 MB/sec execute 108 sec latency 65.625 ms 2024-03-26T01:57:15.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:14 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:57:15.703 INFO:teuthology.orchestra.run.smithi019.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T01:57:15.703 INFO:teuthology.orchestra.run.smithi019.stdout:alertmanager.smithi019 smithi019 *:9093,9094 starting - - - - 2024-03-26T01:57:15.703 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi019 smithi019 running (10m) 92s ago 10m 6405k - 17.2.0 e1d6a67b021e 713db693d5c1 2024-03-26T01:57:15.703 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi069 smithi069 running (9m) 2s ago 9m 6405k - 17.2.0 e1d6a67b021e a87d27e2b74b 2024-03-26T01:57:15.703 INFO:teuthology.orchestra.run.smithi019.stdout:grafana.smithi019 smithi019 *:3000 starting - - - - 2024-03-26T01:57:15.703 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi019.lmryom smithi019 running (4m) 92s ago 4m 36.2M - 17.2.0 e1d6a67b021e 91a7c2942715 2024-03-26T01:57:15.703 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi069.spgwgo smithi069 running (4m) 2s ago 4m 13.3M - 17.2.0 e1d6a67b021e 94d96338dbe1 2024-03-26T01:57:15.703 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi019.ixuvdm smithi019 *:8443,9283,8765 starting - - - - 2024-03-26T01:57:15.704 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi069.wpbewu smithi069 *:8443,9283 running (117s) 2s ago 9m 502M - 19.0.0-1775-g5e70124b 5f21c9264136 c0a5be7a9f9e 2024-03-26T01:57:15.704 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi019 smithi019 running (11m) 92s ago 11m 50.5M 2048M 17.2.0 e1d6a67b021e b6908b497829 2024-03-26T01:57:15.704 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi069 smithi069 running (9m) 2s ago 9m 42.5M 2048M 17.2.0 e1d6a67b021e 2d51b7b7bb32 2024-03-26T01:57:15.704 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.0.0.smithi019.qbizxo smithi019 *:2049 running (4m) 92s ago 4m 265M - 4.0 e1d6a67b021e cd6628a9075d 2024-03-26T01:57:15.704 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.1.0.smithi069.uawdhc smithi069 *:2049 running (3m) 2s ago 3m 83.2M - 4.0 e1d6a67b021e 6ee3695e7a10 2024-03-26T01:57:15.704 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi019 smithi019 *:9100 starting - - - - 2024-03-26T01:57:15.704 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi069 smithi069 *:9100 running (19s) 2s ago 9m 9005k - 1.3.1 1dbe0e931976 c49a006f94e7 2024-03-26T01:57:15.704 INFO:teuthology.orchestra.run.smithi019.stdout:osd.0 smithi019 running (8m) 92s ago 8m 323M 1937M 17.2.0 e1d6a67b021e c1a797230ab6 2024-03-26T01:57:15.704 INFO:teuthology.orchestra.run.smithi019.stdout:osd.1 smithi019 running (8m) 92s ago 8m 248M 1937M 17.2.0 e1d6a67b021e 31d3489c7188 2024-03-26T01:57:15.704 INFO:teuthology.orchestra.run.smithi019.stdout:osd.2 smithi019 running (7m) 92s ago 7m 195M 1937M 17.2.0 e1d6a67b021e f549115e7685 2024-03-26T01:57:15.704 INFO:teuthology.orchestra.run.smithi019.stdout:osd.3 smithi019 running (7m) 92s ago 7m 263M 1937M 17.2.0 e1d6a67b021e 154af68d03e1 2024-03-26T01:57:15.704 INFO:teuthology.orchestra.run.smithi019.stdout:osd.4 smithi069 running (7m) 2s ago 7m 412M 2704M 17.2.0 e1d6a67b021e 4fd5e853f8bf 2024-03-26T01:57:15.704 INFO:teuthology.orchestra.run.smithi019.stdout:osd.5 smithi069 running (6m) 2s ago 6m 461M 2704M 17.2.0 e1d6a67b021e 932e401d87c9 2024-03-26T01:57:15.704 INFO:teuthology.orchestra.run.smithi019.stdout:osd.6 smithi069 running (6m) 2s ago 6m 409M 2704M 17.2.0 e1d6a67b021e b123f75154c8 2024-03-26T01:57:15.704 INFO:teuthology.orchestra.run.smithi019.stdout:osd.7 smithi069 running (5m) 2s ago 5m 538M 2704M 17.2.0 e1d6a67b021e d175fbc9ff8a 2024-03-26T01:57:15.705 INFO:teuthology.orchestra.run.smithi019.stdout:prometheus.smithi019 smithi019 *:9095 starting - - - - 2024-03-26T01:57:15.880 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:15 smithi019 ceph-mon[24816]: from='client.24535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:15.880 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:15 smithi019 ceph-mon[24816]: pgmap v56: 97 pgs: 97 active+clean; 337 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 8.1 MiB/s wr, 815 op/s 2024-03-26T01:57:16.095 INFO:teuthology.orchestra.run.smithi019.stdout: 5 126802 12.70 MB/sec execute 109 sec latency 25.148 ms 2024-03-26T01:57:16.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:15 smithi069 ceph-mon[27071]: from='client.24535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:16.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:15 smithi069 ceph-mon[27071]: pgmap v56: 97 pgs: 97 active+clean; 337 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 8.1 MiB/s wr, 815 op/s 2024-03-26T01:57:16.416 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T01:57:16.416 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": { 2024-03-26T01:57:16.416 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T01:57:16.416 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:57:16.416 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": { 2024-03-26T01:57:16.416 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-26T01:57:16.416 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 1 2024-03-26T01:57:16.416 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:57:16.416 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": { 2024-03-26T01:57:16.417 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T01:57:16.417 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:57:16.417 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": { 2024-03-26T01:57:16.417 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T01:57:16.417 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:57:16.417 INFO:teuthology.orchestra.run.smithi019.stdout: "overall": { 2024-03-26T01:57:16.417 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-26T01:57:16.417 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 1 2024-03-26T01:57:16.417 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-26T01:57:16.417 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T01:57:16.891 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:16 smithi019 ceph-mon[24816]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:16.891 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:16 smithi019 ceph-mon[24816]: from='client.14770 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:16.892 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:16 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/3726216216' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T01:57:16.974 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T01:57:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2", 2024-03-26T01:57:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: "in_progress": true, 2024-03-26T01:57:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T01:57:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: "services_complete": [], 2024-03-26T01:57:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: "progress": "", 2024-03-26T01:57:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: "message": "Currently upgrading mgr daemons", 2024-03-26T01:57:16.974 INFO:teuthology.orchestra.run.smithi019.stdout: "is_paused": false 2024-03-26T01:57:16.974 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T01:57:17.095 INFO:teuthology.orchestra.run.smithi019.stdout: 5 127304 12.77 MB/sec execute 110 sec latency 36.580 ms 2024-03-26T01:57:17.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:16 smithi069 ceph-mon[27071]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:17.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:16 smithi069 ceph-mon[27071]: from='client.14770 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:17.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:16 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/3726216216' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T01:57:17.696 INFO:teuthology.orchestra.run.smithi019.stdout:HEALTH_OK 2024-03-26T01:57:17.896 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:17 smithi019 ceph-mon[24816]: pgmap v57: 97 pgs: 97 active+clean; 323 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 15 KiB/s rd, 8.0 MiB/s wr, 850 op/s 2024-03-26T01:57:18.095 INFO:teuthology.orchestra.run.smithi019.stdout: 5 127703 12.78 MB/sec execute 111 sec latency 33.135 ms 2024-03-26T01:57:18.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:17 smithi069 ceph-mon[27071]: pgmap v57: 97 pgs: 97 active+clean; 323 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 15 KiB/s rd, 8.0 MiB/s wr, 850 op/s 2024-03-26T01:57:18.910 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:18 smithi019 ceph-mon[24816]: from='client.24549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:18.910 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:18 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/2726050200' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T01:57:18.910 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:18 smithi019 ceph-mon[24816]: Standby manager daemon smithi019.ixuvdm restarted 2024-03-26T01:57:18.910 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:18 smithi019 ceph-mon[24816]: Standby manager daemon smithi019.ixuvdm started 2024-03-26T01:57:18.911 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:18 smithi019 ceph-mon[24816]: from='mgr.? 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi019.ixuvdm/crt"}]: dispatch 2024-03-26T01:57:18.911 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:18 smithi019 ceph-mon[24816]: from='mgr.? 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T01:57:18.911 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:18 smithi019 ceph-mon[24816]: from='mgr.? 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi019.ixuvdm/key"}]: dispatch 2024-03-26T01:57:18.911 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:18 smithi019 ceph-mon[24816]: from='mgr.? 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T01:57:18.911 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:18 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:57:18.911 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:18 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:57:19.095 INFO:teuthology.orchestra.run.smithi019.stdout: 5 128093 12.79 MB/sec execute 112 sec latency 39.857 ms 2024-03-26T01:57:19.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:18 smithi069 ceph-mon[27071]: from='client.24549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:19.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:18 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/2726050200' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T01:57:19.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:18 smithi069 ceph-mon[27071]: Standby manager daemon smithi019.ixuvdm restarted 2024-03-26T01:57:19.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:18 smithi069 ceph-mon[27071]: Standby manager daemon smithi019.ixuvdm started 2024-03-26T01:57:19.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:18 smithi069 ceph-mon[27071]: from='mgr.? 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi019.ixuvdm/crt"}]: dispatch 2024-03-26T01:57:19.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:18 smithi069 ceph-mon[27071]: from='mgr.? 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T01:57:19.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:18 smithi069 ceph-mon[27071]: from='mgr.? 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi019.ixuvdm/key"}]: dispatch 2024-03-26T01:57:19.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:18 smithi069 ceph-mon[27071]: from='mgr.? 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T01:57:19.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:18 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:57:19.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:18 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:57:19.921 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:19 smithi019 ceph-mon[24816]: pgmap v58: 97 pgs: 97 active+clean; 323 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 5.9 MiB/s wr, 604 op/s 2024-03-26T01:57:19.921 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:19 smithi019 ceph-mon[24816]: mgrmap e28: smithi069.wpbewu(active, since 111s), standbys: smithi019.ixuvdm 2024-03-26T01:57:19.921 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:19 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:57:19.922 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:19 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:57:20.095 INFO:teuthology.orchestra.run.smithi019.stdout: 5 128326 12.71 MB/sec execute 113 sec latency 39.635 ms 2024-03-26T01:57:20.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:19 smithi069 ceph-mon[27071]: pgmap v58: 97 pgs: 97 active+clean; 323 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 5.9 MiB/s wr, 604 op/s 2024-03-26T01:57:20.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:19 smithi069 ceph-mon[27071]: mgrmap e28: smithi069.wpbewu(active, since 111s), standbys: smithi019.ixuvdm 2024-03-26T01:57:20.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:19 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:57:20.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:19 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:57:21.096 INFO:teuthology.orchestra.run.smithi019.stdout: 5 128605 12.62 MB/sec execute 114 sec latency 45.375 ms 2024-03-26T01:57:21.698 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:21 smithi019 ceph-mon[24816]: pgmap v59: 97 pgs: 97 active+clean; 336 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.8 MiB/s wr, 861 op/s 2024-03-26T01:57:22.096 INFO:teuthology.orchestra.run.smithi019.stdout: 5 128927 12.58 MB/sec execute 115 sec latency 47.842 ms 2024-03-26T01:57:22.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:21 smithi069 ceph-mon[27071]: pgmap v59: 97 pgs: 97 active+clean; 336 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.8 MiB/s wr, 861 op/s 2024-03-26T01:57:23.096 INFO:teuthology.orchestra.run.smithi019.stdout: 5 129300 12.51 MB/sec execute 116 sec latency 41.631 ms 2024-03-26T01:57:23.735 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:23 smithi019 ceph-mon[24816]: pgmap v60: 97 pgs: 97 active+clean; 336 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.5 MiB/s wr, 522 op/s 2024-03-26T01:57:24.000 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:23 smithi069 ceph-mon[27071]: pgmap v60: 97 pgs: 97 active+clean; 336 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.5 MiB/s wr, 522 op/s 2024-03-26T01:57:24.097 INFO:teuthology.orchestra.run.smithi019.stdout: 5 129555 12.49 MB/sec execute 117 sec latency 58.545 ms 2024-03-26T01:57:25.098 INFO:teuthology.orchestra.run.smithi019.stdout: 5 130003 12.48 MB/sec execute 118 sec latency 58.283 ms 2024-03-26T01:57:25.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:25 smithi019 ceph-mon[24816]: pgmap v61: 97 pgs: 97 active+clean; 330 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 6.0 MiB/s wr, 571 op/s 2024-03-26T01:57:26.097 INFO:teuthology.orchestra.run.smithi019.stdout: 5 130434 12.45 MB/sec execute 119 sec latency 29.636 ms 2024-03-26T01:57:26.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:25 smithi069 ceph-mon[27071]: pgmap v61: 97 pgs: 97 active+clean; 330 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 6.0 MiB/s wr, 571 op/s 2024-03-26T01:57:27.041 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:27 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:57:27.041 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:27 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:57:27.041 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:27 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:27.042 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:27 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:27.042 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:27 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:57:27.042 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:27 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:57:27.042 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:27 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:27.042 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:27 smithi019 ceph-mon[24816]: Upgrade: Need to upgrade myself (mgr.smithi069.wpbewu) 2024-03-26T01:57:27.042 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:27 smithi019 ceph-mon[24816]: Upgrade: Need to upgrade myself (mgr.smithi069.wpbewu) 2024-03-26T01:57:27.042 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:27 smithi019 ceph-mon[24816]: Failing over to other MGR 2024-03-26T01:57:27.042 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:27 smithi019 ceph-mon[24816]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "mgr fail", "who": "smithi069.wpbewu"}]: dispatch 2024-03-26T01:57:27.042 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:27 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "mgr fail", "who": "smithi069.wpbewu"}]: dispatch 2024-03-26T01:57:27.042 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:27 smithi019 ceph-mon[24816]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T01:57:27.042 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:27 smithi019 ceph-mon[24816]: Standby manager daemon smithi069.wpbewu started 2024-03-26T01:57:27.043 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:27 smithi019 ceph-mon[24816]: pgmap v63: 97 pgs: 97 active+clean; 323 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.5 MiB/s wr, 553 op/s 2024-03-26T01:57:27.098 INFO:teuthology.orchestra.run.smithi019.stdout: 5 130811 12.49 MB/sec execute 120 sec latency 48.222 ms 2024-03-26T01:57:27.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:27 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:57:27.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:27 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:57:27.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:27 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:27.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:27 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:27.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:27 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:57:27.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:27 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' 2024-03-26T01:57:27.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:27 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:27.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:27 smithi069 ceph-mon[27071]: Upgrade: Need to upgrade myself (mgr.smithi069.wpbewu) 2024-03-26T01:57:27.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:27 smithi069 ceph-mon[27071]: Upgrade: Need to upgrade myself (mgr.smithi069.wpbewu) 2024-03-26T01:57:27.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:27 smithi069 ceph-mon[27071]: Failing over to other MGR 2024-03-26T01:57:27.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:27 smithi069 ceph-mon[27071]: from='mgr.24431 172.21.15.69:0/3623243131' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "mgr fail", "who": "smithi069.wpbewu"}]: dispatch 2024-03-26T01:57:27.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:27 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "mgr fail", "who": "smithi069.wpbewu"}]: dispatch 2024-03-26T01:57:27.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:27 smithi069 ceph-mon[27071]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T01:57:27.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:27 smithi069 ceph-mon[27071]: Standby manager daemon smithi069.wpbewu started 2024-03-26T01:57:27.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:27 smithi069 ceph-mon[27071]: pgmap v63: 97 pgs: 97 active+clean; 323 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.5 MiB/s wr, 553 op/s 2024-03-26T01:57:28.098 INFO:teuthology.orchestra.run.smithi019.stdout: 5 131124 12.48 MB/sec execute 121 sec latency 42.488 ms 2024-03-26T01:57:28.319 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:28 smithi019 ceph-mon[24816]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' cmd='[{"prefix": "mgr fail", "who": "smithi069.wpbewu"}]': finished 2024-03-26T01:57:28.319 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:28 smithi019 ceph-mon[24816]: mgrmap e29: smithi019.ixuvdm(active, starting, since 0.857988s), standbys: smithi069.wpbewu 2024-03-26T01:57:28.319 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:28 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi019"}]: dispatch 2024-03-26T01:57:28.319 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:28 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi069"}]: dispatch 2024-03-26T01:57:28.319 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:28 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi019.lmryom"}]: dispatch 2024-03-26T01:57:28.319 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:28 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi069.spgwgo"}]: dispatch 2024-03-26T01:57:28.319 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:28 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mgr metadata", "who": "smithi019.ixuvdm", "id": "smithi019.ixuvdm"}]: dispatch 2024-03-26T01:57:28.319 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:28 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mgr metadata", "who": "smithi069.wpbewu", "id": "smithi069.wpbewu"}]: dispatch 2024-03-26T01:57:28.319 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:28 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:28.320 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:28 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:57:28.320 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:28 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:57:28.320 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:28 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:57:28.320 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:28 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T01:57:28.320 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:28 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T01:57:28.320 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:28 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T01:57:28.320 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:28 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T01:57:28.320 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:28 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T01:57:28.320 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:28 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T01:57:28.320 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:28 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T01:57:28.320 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:28 smithi019 ceph-mon[24816]: Manager daemon smithi019.ixuvdm is now available 2024-03-26T01:57:28.320 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:28 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:28.320 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:28 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:28.320 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:28 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:57:28.320 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:28 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:57:28.321 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:28 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/trash_purge_schedule"}]: dispatch 2024-03-26T01:57:28.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:28 smithi069 ceph-mon[27071]: from='mgr.24431 ' entity='mgr.smithi069.wpbewu' cmd='[{"prefix": "mgr fail", "who": "smithi069.wpbewu"}]': finished 2024-03-26T01:57:28.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:28 smithi069 ceph-mon[27071]: mgrmap e29: smithi019.ixuvdm(active, starting, since 0.857988s), standbys: smithi069.wpbewu 2024-03-26T01:57:28.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:28 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi019"}]: dispatch 2024-03-26T01:57:28.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:28 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi069"}]: dispatch 2024-03-26T01:57:28.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:28 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi019.lmryom"}]: dispatch 2024-03-26T01:57:28.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:28 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi069.spgwgo"}]: dispatch 2024-03-26T01:57:28.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:28 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mgr metadata", "who": "smithi019.ixuvdm", "id": "smithi019.ixuvdm"}]: dispatch 2024-03-26T01:57:28.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:28 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mgr metadata", "who": "smithi069.wpbewu", "id": "smithi069.wpbewu"}]: dispatch 2024-03-26T01:57:28.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:28 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:28.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:28 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:57:28.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:28 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:57:28.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:28 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:57:28.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:28 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T01:57:28.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:28 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T01:57:28.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:28 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T01:57:28.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:28 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T01:57:28.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:28 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T01:57:28.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:28 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T01:57:28.361 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:28 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T01:57:28.361 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:28 smithi069 ceph-mon[27071]: Manager daemon smithi019.ixuvdm is now available 2024-03-26T01:57:28.361 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:28 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:28.361 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:28 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:28.361 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:28 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:57:28.361 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:28 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:57:28.361 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:28 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/trash_purge_schedule"}]: dispatch 2024-03-26T01:57:29.098 INFO:teuthology.orchestra.run.smithi019.stdout: 5 131562 12.48 MB/sec execute 122 sec latency 24.732 ms 2024-03-26T01:57:29.327 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:29 smithi019 ceph-mon[24816]: [26/Mar/2024:01:57:28] ENGINE Bus STARTING 2024-03-26T01:57:29.327 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:29 smithi019 ceph-mon[24816]: mgrmap e30: smithi019.ixuvdm(active, since 1.87403s), standbys: smithi069.wpbewu 2024-03-26T01:57:29.327 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:29 smithi019 ceph-mon[24816]: pgmap v3: 97 pgs: 97 active+clean; 335 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-03-26T01:57:29.327 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:29 smithi019 ceph-mon[24816]: [26/Mar/2024:01:57:28] ENGINE Serving on http://172.21.15.19:8765 2024-03-26T01:57:29.328 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:29 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:29.328 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:29 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:29.328 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:29 smithi019 ceph-mon[24816]: [26/Mar/2024:01:57:28] ENGINE Serving on https://172.21.15.19:7150 2024-03-26T01:57:29.328 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:29 smithi019 ceph-mon[24816]: [26/Mar/2024:01:57:28] ENGINE Bus STARTED 2024-03-26T01:57:29.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:29 smithi069 ceph-mon[27071]: [26/Mar/2024:01:57:28] ENGINE Bus STARTING 2024-03-26T01:57:29.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:29 smithi069 ceph-mon[27071]: mgrmap e30: smithi019.ixuvdm(active, since 1.87403s), standbys: smithi069.wpbewu 2024-03-26T01:57:29.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:29 smithi069 ceph-mon[27071]: pgmap v3: 97 pgs: 97 active+clean; 335 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-03-26T01:57:29.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:29 smithi069 ceph-mon[27071]: [26/Mar/2024:01:57:28] ENGINE Serving on http://172.21.15.19:8765 2024-03-26T01:57:29.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:29 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:29.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:29 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:29.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:29 smithi069 ceph-mon[27071]: [26/Mar/2024:01:57:28] ENGINE Serving on https://172.21.15.19:7150 2024-03-26T01:57:29.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:29 smithi069 ceph-mon[27071]: [26/Mar/2024:01:57:28] ENGINE Bus STARTED 2024-03-26T01:57:30.098 INFO:teuthology.orchestra.run.smithi019.stdout: 5 131882 12.45 MB/sec execute 123 sec latency 31.793 ms 2024-03-26T01:57:30.170 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:30 smithi069 ceph-mon[27071]: pgmap v4: 97 pgs: 97 active+clean; 335 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-03-26T01:57:30.342 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:30 smithi019 ceph-mon[24816]: pgmap v4: 97 pgs: 97 active+clean; 335 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-03-26T01:57:31.099 INFO:teuthology.orchestra.run.smithi019.stdout: 5 132124 12.37 MB/sec execute 124 sec latency 62.574 ms 2024-03-26T01:57:31.355 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:31 smithi019 ceph-mon[24816]: mgrmap e31: smithi019.ixuvdm(active, since 3s), standbys: smithi069.wpbewu 2024-03-26T01:57:31.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:31 smithi069 ceph-mon[27071]: mgrmap e31: smithi019.ixuvdm(active, since 3s), standbys: smithi069.wpbewu 2024-03-26T01:57:32.098 INFO:teuthology.orchestra.run.smithi019.stdout: 5 132379 12.33 MB/sec execute 125 sec latency 65.853 ms 2024-03-26T01:57:32.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:32 smithi069 ceph-mon[27071]: pgmap v5: 97 pgs: 97 active+clean; 335 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-03-26T01:57:32.366 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:32 smithi019 ceph-mon[24816]: pgmap v5: 97 pgs: 97 active+clean; 335 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-03-26T01:57:33.098 INFO:teuthology.orchestra.run.smithi019.stdout: 5 132640 12.26 MB/sec execute 126 sec latency 67.000 ms 2024-03-26T01:57:33.657 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:33 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:33.657 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:33 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:33.847 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:33 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:33.848 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:33 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:34.098 INFO:teuthology.orchestra.run.smithi019.stdout: 5 132897 12.19 MB/sec execute 127 sec latency 74.731 ms 2024-03-26T01:57:34.957 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:34 smithi019 ceph-mon[24816]: pgmap v6: 97 pgs: 97 active+clean; 335 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-03-26T01:57:34.957 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:34 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:34.957 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:34 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:35.100 INFO:teuthology.orchestra.run.smithi019.stdout: 5 133227 12.17 MB/sec execute 128 sec latency 53.904 ms 2024-03-26T01:57:35.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:34 smithi069 ceph-mon[27071]: pgmap v6: 97 pgs: 97 active+clean; 335 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-03-26T01:57:35.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:34 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:35.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:34 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:36.100 INFO:teuthology.orchestra.run.smithi019.stdout: 5 133700 12.19 MB/sec execute 129 sec latency 41.086 ms 2024-03-26T01:57:37.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:36 smithi019 ceph-mon[24816]: pgmap v7: 97 pgs: 97 active+clean; 335 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 36 KiB/s rd, 3.3 MiB/s wr, 357 op/s 2024-03-26T01:57:37.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:36 smithi019 ceph-mon[24816]: from='mgr.? 172.21.15.69:0/572374563' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi069.wpbewu/crt"}]: dispatch 2024-03-26T01:57:37.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:36 smithi019 ceph-mon[24816]: from='mgr.? 172.21.15.69:0/572374563' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T01:57:37.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:36 smithi019 ceph-mon[24816]: from='mgr.? 172.21.15.69:0/572374563' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi069.wpbewu/key"}]: dispatch 2024-03-26T01:57:37.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:36 smithi019 ceph-mon[24816]: from='mgr.? 172.21.15.69:0/572374563' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T01:57:37.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:36 smithi019 ceph-mon[24816]: Standby manager daemon smithi069.wpbewu restarted 2024-03-26T01:57:37.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:36 smithi019 ceph-mon[24816]: Standby manager daemon smithi069.wpbewu started 2024-03-26T01:57:37.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:36 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:37.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:36 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:37.100 INFO:teuthology.orchestra.run.smithi019.stdout: 5 134040 12.15 MB/sec execute 130 sec latency 36.987 ms 2024-03-26T01:57:37.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:36 smithi069 ceph-mon[27071]: pgmap v7: 97 pgs: 97 active+clean; 335 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 36 KiB/s rd, 3.3 MiB/s wr, 357 op/s 2024-03-26T01:57:37.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:36 smithi069 ceph-mon[27071]: from='mgr.? 172.21.15.69:0/572374563' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi069.wpbewu/crt"}]: dispatch 2024-03-26T01:57:37.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:36 smithi069 ceph-mon[27071]: from='mgr.? 172.21.15.69:0/572374563' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T01:57:37.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:36 smithi069 ceph-mon[27071]: from='mgr.? 172.21.15.69:0/572374563' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi069.wpbewu/key"}]: dispatch 2024-03-26T01:57:37.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:36 smithi069 ceph-mon[27071]: from='mgr.? 172.21.15.69:0/572374563' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T01:57:37.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:36 smithi069 ceph-mon[27071]: Standby manager daemon smithi069.wpbewu restarted 2024-03-26T01:57:37.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:36 smithi069 ceph-mon[27071]: Standby manager daemon smithi069.wpbewu started 2024-03-26T01:57:37.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:36 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:37.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:36 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:37.866 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:37 smithi019 ceph-mon[24816]: mgrmap e32: smithi019.ixuvdm(active, since 10s), standbys: smithi069.wpbewu 2024-03-26T01:57:37.866 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:37 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:37.866 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:37 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:38.100 INFO:teuthology.orchestra.run.smithi019.stdout: 5 134418 12.20 MB/sec execute 131 sec latency 37.186 ms 2024-03-26T01:57:38.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:37 smithi069 ceph-mon[27071]: mgrmap e32: smithi019.ixuvdm(active, since 10s), standbys: smithi069.wpbewu 2024-03-26T01:57:38.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:37 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:38.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:37 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:38.889 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:38 smithi019 ceph-mon[24816]: pgmap v8: 97 pgs: 97 active+clean; 335 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 28 KiB/s rd, 2.6 MiB/s wr, 277 op/s 2024-03-26T01:57:39.100 INFO:teuthology.orchestra.run.smithi019.stdout: 5 134779 12.19 MB/sec execute 132 sec latency 37.669 ms 2024-03-26T01:57:39.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:38 smithi069 ceph-mon[27071]: pgmap v8: 97 pgs: 97 active+clean; 335 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 28 KiB/s rd, 2.6 MiB/s wr, 277 op/s 2024-03-26T01:57:40.100 INFO:teuthology.orchestra.run.smithi019.stdout: 5 135160 12.18 MB/sec execute 133 sec latency 37.210 ms 2024-03-26T01:57:40.908 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:40 smithi019 ceph-mon[24816]: pgmap v9: 97 pgs: 97 active+clean; 333 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 4.5 MiB/s wr, 459 op/s 2024-03-26T01:57:41.100 INFO:teuthology.orchestra.run.smithi019.stdout: 5 135413 12.14 MB/sec execute 134 sec latency 43.415 ms 2024-03-26T01:57:41.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:40 smithi069 ceph-mon[27071]: pgmap v9: 97 pgs: 97 active+clean; 333 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 4.5 MiB/s wr, 459 op/s 2024-03-26T01:57:42.100 INFO:teuthology.orchestra.run.smithi019.stdout: 5 135646 12.07 MB/sec execute 135 sec latency 50.250 ms 2024-03-26T01:57:42.965 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:42 smithi019 ceph-mon[24816]: pgmap v10: 97 pgs: 97 active+clean; 333 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 4.1 MiB/s wr, 418 op/s 2024-03-26T01:57:42.965 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:42 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:57:43.100 INFO:teuthology.orchestra.run.smithi019.stdout: 5 135890 12.04 MB/sec execute 136 sec latency 76.920 ms 2024-03-26T01:57:43.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:42 smithi069 ceph-mon[27071]: pgmap v10: 97 pgs: 97 active+clean; 333 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 4.1 MiB/s wr, 418 op/s 2024-03-26T01:57:43.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:42 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:57:44.100 INFO:teuthology.orchestra.run.smithi019.stdout: 5 136168 11.97 MB/sec execute 137 sec latency 74.935 ms 2024-03-26T01:57:44.490 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:44 smithi019 ceph-mon[24816]: pgmap v11: 97 pgs: 97 active+clean; 333 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 4.1 MiB/s wr, 419 op/s 2024-03-26T01:57:44.490 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:44 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:44.490 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:44 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:44.490 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:44 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:44.491 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:44 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:44.491 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:44 smithi019 ceph-mon[24816]: Updating smithi019:/etc/ceph/ceph.conf 2024-03-26T01:57:44.491 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:44 smithi019 ceph-mon[24816]: Updating smithi069:/etc/ceph/ceph.conf 2024-03-26T01:57:44.491 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:44 smithi019 ceph-mon[24816]: Updating smithi019:/var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:57:44.491 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:44 smithi019 ceph-mon[24816]: Updating smithi069:/var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:57:44.491 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:44 smithi019 ceph-mon[24816]: Updating smithi019:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:57:44.491 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:44 smithi019 ceph-mon[24816]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:57:44.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:44 smithi069 ceph-mon[27071]: pgmap v11: 97 pgs: 97 active+clean; 333 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 4.1 MiB/s wr, 419 op/s 2024-03-26T01:57:44.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:44 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:44.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:44 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:44.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:44 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:44.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:44 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:44.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:44 smithi069 ceph-mon[27071]: Updating smithi019:/etc/ceph/ceph.conf 2024-03-26T01:57:44.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:44 smithi069 ceph-mon[27071]: Updating smithi069:/etc/ceph/ceph.conf 2024-03-26T01:57:44.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:44 smithi069 ceph-mon[27071]: Updating smithi019:/var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:57:44.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:44 smithi069 ceph-mon[27071]: Updating smithi069:/var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:57:44.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:44 smithi069 ceph-mon[27071]: Updating smithi019:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:57:44.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:44 smithi069 ceph-mon[27071]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:57:45.102 INFO:teuthology.orchestra.run.smithi019.stdout: 5 136562 11.97 MB/sec execute 138 sec latency 40.135 ms 2024-03-26T01:57:45.507 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:45 smithi019 ceph-mon[24816]: Updating smithi019:/var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T01:57:45.507 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:45 smithi019 ceph-mon[24816]: Updating smithi069:/var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T01:57:45.507 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:45 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:45.507 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:45 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:45.507 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:45 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:45.508 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:45 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:45.508 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:45 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:45.508 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:45 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:45.508 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:45 smithi019 ceph-mon[24816]: Reconfiguring prometheus.smithi019 (dependencies changed)... 2024-03-26T01:57:45.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:45 smithi069 ceph-mon[27071]: Updating smithi019:/var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T01:57:45.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:45 smithi069 ceph-mon[27071]: Updating smithi069:/var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T01:57:45.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:45 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:45.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:45 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:45.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:45 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:45.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:45 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:45.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:45 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:45.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:45 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:45.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:45 smithi069 ceph-mon[27071]: Reconfiguring prometheus.smithi019 (dependencies changed)... 2024-03-26T01:57:46.100 INFO:teuthology.orchestra.run.smithi019.stdout: 5 136988 11.95 MB/sec execute 139 sec latency 46.770 ms 2024-03-26T01:57:46.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:46 smithi019 ceph-mon[24816]: Reconfiguring daemon prometheus.smithi019 on smithi019 2024-03-26T01:57:46.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:46 smithi019 ceph-mon[24816]: pgmap v12: 97 pgs: 97 active+clean; 336 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 6.0 MiB/s wr, 607 op/s 2024-03-26T01:57:46.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:46 smithi069 ceph-mon[27071]: Reconfiguring daemon prometheus.smithi019 on smithi019 2024-03-26T01:57:46.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:46 smithi069 ceph-mon[27071]: pgmap v12: 97 pgs: 97 active+clean; 336 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 6.0 MiB/s wr, 607 op/s 2024-03-26T01:57:47.100 INFO:teuthology.orchestra.run.smithi019.stdout: 5 137492 11.95 MB/sec execute 140 sec latency 32.655 ms 2024-03-26T01:57:48.100 INFO:teuthology.orchestra.run.smithi019.stdout: 5 137831 11.96 MB/sec execute 141 sec latency 40.508 ms 2024-03-26T01:57:48.181 INFO:teuthology.orchestra.run.smithi019.stdout:true 2024-03-26T01:57:49.076 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:48 smithi019 ceph-mon[24816]: pgmap v13: 97 pgs: 97 active+clean; 336 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.1 MiB/s wr, 400 op/s 2024-03-26T01:57:49.085 INFO:teuthology.orchestra.run.smithi019.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T01:57:49.085 INFO:teuthology.orchestra.run.smithi019.stdout:alertmanager.smithi019 smithi019 *:9093,9094 running (108s) 12s ago 10m 16.2M - 0.23.0 ba2b418f427c a6630841c2b2 2024-03-26T01:57:49.085 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi019 smithi019 running (10m) 12s ago 10m 6405k - 17.2.0 e1d6a67b021e 713db693d5c1 2024-03-26T01:57:49.085 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi069 smithi069 running (10m) 16s ago 10m 6405k - 17.2.0 e1d6a67b021e a87d27e2b74b 2024-03-26T01:57:49.085 INFO:teuthology.orchestra.run.smithi019.stdout:grafana.smithi019 smithi019 *:3000 running (84s) 12s ago 10m 46.8M - 8.3.5 dad864ee21e9 a97dea3b5fec 2024-03-26T01:57:49.085 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi019.lmryom smithi019 running (4m) 12s ago 4m 37.2M - 17.2.0 e1d6a67b021e 91a7c2942715 2024-03-26T01:57:49.085 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi069.spgwgo smithi069 running (4m) 16s ago 4m 13.6M - 17.2.0 e1d6a67b021e 94d96338dbe1 2024-03-26T01:57:49.085 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi019.ixuvdm smithi019 *:8443,9283,8765 running (41s) 12s ago 12m 481M - 19.0.0-1775-g5e70124b 5f21c9264136 4460b22c7a56 2024-03-26T01:57:49.085 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi069.wpbewu smithi069 *:8443,9283 running (2m) 16s ago 10m 105M - 19.0.0-1775-g5e70124b 5f21c9264136 c0a5be7a9f9e 2024-03-26T01:57:49.085 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi019 smithi019 running (12m) 12s ago 12m 57.0M 2048M 17.2.0 e1d6a67b021e b6908b497829 2024-03-26T01:57:49.085 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi069 smithi069 running (10m) 16s ago 10m 45.2M 2048M 17.2.0 e1d6a67b021e 2d51b7b7bb32 2024-03-26T01:57:49.085 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.0.0.smithi019.qbizxo smithi019 *:2049 running (4m) 12s ago 4m 280M - 4.0 e1d6a67b021e cd6628a9075d 2024-03-26T01:57:49.086 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.1.0.smithi069.uawdhc smithi069 *:2049 running (4m) 16s ago 4m 83.5M - 4.0 e1d6a67b021e 6ee3695e7a10 2024-03-26T01:57:49.086 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi019 smithi019 *:9100 running (81s) 12s ago 10m 9013k - 1.3.1 1dbe0e931976 0d1d184bec63 2024-03-26T01:57:49.086 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi069 smithi069 *:9100 running (53s) 16s ago 10m 10.4M - 1.3.1 1dbe0e931976 c49a006f94e7 2024-03-26T01:57:49.086 INFO:teuthology.orchestra.run.smithi019.stdout:osd.0 smithi019 running (9m) 12s ago 9m 466M 1937M 17.2.0 e1d6a67b021e c1a797230ab6 2024-03-26T01:57:49.086 INFO:teuthology.orchestra.run.smithi019.stdout:osd.1 smithi019 running (8m) 12s ago 8m 405M 1937M 17.2.0 e1d6a67b021e 31d3489c7188 2024-03-26T01:57:49.086 INFO:teuthology.orchestra.run.smithi019.stdout:osd.2 smithi019 running (8m) 12s ago 8m 283M 1937M 17.2.0 e1d6a67b021e f549115e7685 2024-03-26T01:57:49.086 INFO:teuthology.orchestra.run.smithi019.stdout:osd.3 smithi019 running (7m) 12s ago 7m 425M 1937M 17.2.0 e1d6a67b021e 154af68d03e1 2024-03-26T01:57:49.086 INFO:teuthology.orchestra.run.smithi019.stdout:osd.4 smithi069 running (7m) 16s ago 7m 446M 2704M 17.2.0 e1d6a67b021e 4fd5e853f8bf 2024-03-26T01:57:49.086 INFO:teuthology.orchestra.run.smithi019.stdout:osd.5 smithi069 running (7m) 16s ago 7m 497M 2704M 17.2.0 e1d6a67b021e 932e401d87c9 2024-03-26T01:57:49.086 INFO:teuthology.orchestra.run.smithi019.stdout:osd.6 smithi069 running (6m) 16s ago 6m 420M 2704M 17.2.0 e1d6a67b021e b123f75154c8 2024-03-26T01:57:49.086 INFO:teuthology.orchestra.run.smithi019.stdout:osd.7 smithi069 running (6m) 16s ago 6m 586M 2704M 17.2.0 e1d6a67b021e d175fbc9ff8a 2024-03-26T01:57:49.086 INFO:teuthology.orchestra.run.smithi019.stdout:prometheus.smithi019 smithi019 *:9095 running (55s) 12s ago 9m 50.0M - 2.33.4 514e6a882f6e 798e9f444f8c 2024-03-26T01:57:49.100 INFO:teuthology.orchestra.run.smithi019.stdout: 5 138108 11.95 MB/sec execute 142 sec latency 42.820 ms 2024-03-26T01:57:49.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:48 smithi069 ceph-mon[27071]: pgmap v13: 97 pgs: 97 active+clean; 336 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.1 MiB/s wr, 400 op/s 2024-03-26T01:57:49.629 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T01:57:49.629 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": { 2024-03-26T01:57:49.629 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T01:57:49.629 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:57:49.629 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": { 2024-03-26T01:57:49.629 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T01:57:49.629 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:57:49.630 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": { 2024-03-26T01:57:49.630 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T01:57:49.630 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:57:49.630 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": { 2024-03-26T01:57:49.630 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T01:57:49.630 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:57:49.630 INFO:teuthology.orchestra.run.smithi019.stdout: "overall": { 2024-03-26T01:57:49.630 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-03-26T01:57:49.630 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T01:57:49.630 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-26T01:57:49.630 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T01:57:49.858 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:49 smithi019 ceph-mon[24816]: from='client.14812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:49.858 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:49 smithi019 ceph-mon[24816]: from='client.14816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:49.858 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:49 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/2734536827' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T01:57:50.094 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T01:57:50.095 INFO:teuthology.orchestra.run.smithi019.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2", 2024-03-26T01:57:50.095 INFO:teuthology.orchestra.run.smithi019.stdout: "in_progress": true, 2024-03-26T01:57:50.095 INFO:teuthology.orchestra.run.smithi019.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T01:57:50.095 INFO:teuthology.orchestra.run.smithi019.stdout: "services_complete": [ 2024-03-26T01:57:50.095 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr" 2024-03-26T01:57:50.095 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-26T01:57:50.095 INFO:teuthology.orchestra.run.smithi019.stdout: "progress": "2/23 daemons upgraded", 2024-03-26T01:57:50.095 INFO:teuthology.orchestra.run.smithi019.stdout: "message": "", 2024-03-26T01:57:50.095 INFO:teuthology.orchestra.run.smithi019.stdout: "is_paused": false 2024-03-26T01:57:50.095 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T01:57:50.100 INFO:teuthology.orchestra.run.smithi019.stdout: 5 138417 11.95 MB/sec execute 143 sec latency 39.983 ms 2024-03-26T01:57:50.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:49 smithi069 ceph-mon[27071]: from='client.14812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:50.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:49 smithi069 ceph-mon[27071]: from='client.14816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:50.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:49 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/2734536827' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T01:57:50.669 INFO:teuthology.orchestra.run.smithi019.stdout:HEALTH_OK 2024-03-26T01:57:50.885 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:50 smithi019 ceph-mon[24816]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:50.885 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:50 smithi019 ceph-mon[24816]: pgmap v14: 97 pgs: 97 active+clean; 327 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 6.2 MiB/s wr, 625 op/s 2024-03-26T01:57:50.885 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:50 smithi019 ceph-mon[24816]: from='client.? 172.21.15.19:0/1959117598' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T01:57:51.100 INFO:teuthology.orchestra.run.smithi019.stdout: 5 138749 11.92 MB/sec execute 144 sec latency 39.895 ms 2024-03-26T01:57:51.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:50 smithi069 ceph-mon[27071]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:51.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:50 smithi069 ceph-mon[27071]: pgmap v14: 97 pgs: 97 active+clean; 327 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 6.2 MiB/s wr, 625 op/s 2024-03-26T01:57:51.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:50 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/1959117598' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T01:57:51.892 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:51 smithi019 ceph-mon[24816]: from='client.14828 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:52.100 INFO:teuthology.orchestra.run.smithi019.stdout: 5 139033 11.89 MB/sec execute 145 sec latency 29.169 ms 2024-03-26T01:57:52.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:51 smithi069 ceph-mon[27071]: from='client.14828 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:52.915 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:52 smithi019 ceph-mon[24816]: pgmap v15: 97 pgs: 97 active+clean; 327 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.1 MiB/s wr, 413 op/s 2024-03-26T01:57:53.101 INFO:teuthology.orchestra.run.smithi019.stdout: 5 139368 11.86 MB/sec execute 146 sec latency 57.765 ms 2024-03-26T01:57:53.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:52 smithi069 ceph-mon[27071]: pgmap v15: 97 pgs: 97 active+clean; 327 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.1 MiB/s wr, 413 op/s 2024-03-26T01:57:54.100 INFO:teuthology.orchestra.run.smithi019.stdout: 5 139827 11.81 MB/sec execute 147 sec latency 56.017 ms 2024-03-26T01:57:54.947 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:54 smithi019 ceph-mon[24816]: pgmap v16: 97 pgs: 97 active+clean; 327 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.1 MiB/s wr, 413 op/s 2024-03-26T01:57:54.948 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:54 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:54.948 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:54 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:54.948 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:54 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T01:57:54.948 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:54 smithi019 ceph-mon[24816]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T01:57:54.948 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:54 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:55.101 INFO:teuthology.orchestra.run.smithi019.stdout: 5 140320 11.82 MB/sec execute 148 sec latency 28.386 ms 2024-03-26T01:57:55.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:54 smithi069 ceph-mon[27071]: pgmap v16: 97 pgs: 97 active+clean; 327 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.1 MiB/s wr, 413 op/s 2024-03-26T01:57:55.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:54 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:55.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:54 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:57:55.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:54 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T01:57:55.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:54 smithi069 ceph-mon[27071]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T01:57:55.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:54 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:56.101 INFO:teuthology.orchestra.run.smithi019.stdout: 5 141010 11.87 MB/sec execute 149 sec latency 26.944 ms 2024-03-26T01:57:56.957 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:56 smithi019 ceph-mon[24816]: pgmap v17: 97 pgs: 97 active+clean; 337 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 6.3 MiB/s wr, 603 op/s 2024-03-26T01:57:56.957 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:56 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi069.wpbewu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T01:57:56.957 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:56 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T01:57:56.957 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:56 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:56.969 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:56 smithi069 ceph-mon[27071]: pgmap v17: 97 pgs: 97 active+clean; 337 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 6.3 MiB/s wr, 603 op/s 2024-03-26T01:57:56.969 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:56 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi069.wpbewu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T01:57:56.969 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:56 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T01:57:56.969 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:56 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:57.101 INFO:teuthology.orchestra.run.smithi019.stdout: 5 141656 11.95 MB/sec execute 150 sec latency 30.102 ms 2024-03-26T01:57:57.971 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:57 smithi019 ceph-mon[24816]: Upgrade: Updating mgr.smithi069.wpbewu 2024-03-26T01:57:57.971 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:57 smithi019 ceph-mon[24816]: Deploying daemon mgr.smithi069.wpbewu on smithi069 2024-03-26T01:57:57.972 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:57 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:57:58.101 INFO:teuthology.orchestra.run.smithi019.stdout: 5 142398 12.03 MB/sec execute 151 sec latency 38.669 ms 2024-03-26T01:57:58.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:57 smithi069 ceph-mon[27071]: Upgrade: Updating mgr.smithi069.wpbewu 2024-03-26T01:57:58.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:57 smithi069 ceph-mon[27071]: Deploying daemon mgr.smithi069.wpbewu on smithi069 2024-03-26T01:57:58.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:57 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:57:58.979 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:57:58 smithi019 ceph-mon[24816]: pgmap v18: 97 pgs: 97 active+clean; 337 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.3 MiB/s wr, 415 op/s 2024-03-26T01:57:59.101 INFO:teuthology.orchestra.run.smithi019.stdout: 5 142975 12.03 MB/sec execute 152 sec latency 20.931 ms 2024-03-26T01:57:59.282 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:57:58 smithi069 ceph-mon[27071]: pgmap v18: 97 pgs: 97 active+clean; 337 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.3 MiB/s wr, 415 op/s 2024-03-26T01:58:00.101 INFO:teuthology.orchestra.run.smithi019.stdout: 5 143665 12.03 MB/sec execute 153 sec latency 36.976 ms 2024-03-26T01:58:00.994 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:00 smithi019 ceph-mon[24816]: pgmap v19: 97 pgs: 97 active+clean; 339 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.8 KiB/s rd, 8.0 MiB/s wr, 773 op/s 2024-03-26T01:58:00.994 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:00 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:00.994 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:00 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:00.994 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:00 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:01.101 INFO:teuthology.orchestra.run.smithi019.stdout: 5 144346 12.08 MB/sec execute 154 sec latency 32.316 ms 2024-03-26T01:58:01.159 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:00 smithi069 ceph-mon[27071]: pgmap v19: 97 pgs: 97 active+clean; 339 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.8 KiB/s rd, 8.0 MiB/s wr, 773 op/s 2024-03-26T01:58:01.159 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:00 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:01.160 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:00 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:01.160 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:00 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:02.101 INFO:teuthology.orchestra.run.smithi019.stdout: 5 144911 12.12 MB/sec execute 155 sec latency 43.845 ms 2024-03-26T01:58:03.076 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:02 smithi019 ceph-mon[24816]: pgmap v20: 97 pgs: 97 active+clean; 339 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.9 MiB/s wr, 548 op/s 2024-03-26T01:58:03.101 INFO:teuthology.orchestra.run.smithi019.stdout: 5 145238 12.11 MB/sec execute 156 sec latency 61.690 ms 2024-03-26T01:58:03.205 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:02 smithi069 ceph-mon[27071]: pgmap v20: 97 pgs: 97 active+clean; 339 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.9 MiB/s wr, 548 op/s 2024-03-26T01:58:04.101 INFO:teuthology.orchestra.run.smithi019.stdout: 5 145665 12.11 MB/sec execute 157 sec latency 38.785 ms 2024-03-26T01:58:05.052 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:04 smithi019 ceph-mon[24816]: pgmap v21: 97 pgs: 97 active+clean; 338 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 6.3 MiB/s wr, 613 op/s 2024-03-26T01:58:05.073 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:04 smithi069 ceph-mon[27071]: pgmap v21: 97 pgs: 97 active+clean; 338 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 6.3 MiB/s wr, 613 op/s 2024-03-26T01:58:05.101 INFO:teuthology.orchestra.run.smithi019.stdout: 5 146055 12.11 MB/sec execute 158 sec latency 30.191 ms 2024-03-26T01:58:06.101 INFO:teuthology.orchestra.run.smithi019.stdout: 5 146413 12.10 MB/sec execute 159 sec latency 39.554 ms 2024-03-26T01:58:06.318 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:06 smithi019 ceph-mon[24816]: pgmap v22: 97 pgs: 97 active+clean; 337 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 9.3 MiB/s wr, 883 op/s 2024-03-26T01:58:06.318 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:06 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:06.318 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:06 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:06.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:06 smithi069 ceph-mon[27071]: pgmap v22: 97 pgs: 97 active+clean; 337 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 9.3 MiB/s wr, 883 op/s 2024-03-26T01:58:06.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:06 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:06.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:06 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:07.101 INFO:teuthology.orchestra.run.smithi019.stdout: 5 146907 12.05 MB/sec execute 160 sec latency 43.334 ms 2024-03-26T01:58:07.586 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:07 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:07.586 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:07 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:07.704 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:07 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:07.704 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:07 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:08.101 INFO:teuthology.orchestra.run.smithi019.stdout: 5 147375 12.05 MB/sec execute 161 sec latency 30.856 ms 2024-03-26T01:58:08.856 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:08 smithi019 ceph-mon[24816]: pgmap v23: 97 pgs: 97 active+clean; 337 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.1 MiB/s wr, 692 op/s 2024-03-26T01:58:08.972 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:08 smithi069 ceph-mon[27071]: pgmap v23: 97 pgs: 97 active+clean; 337 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.1 MiB/s wr, 692 op/s 2024-03-26T01:58:09.101 INFO:teuthology.orchestra.run.smithi019.stdout: 5 147930 12.08 MB/sec execute 162 sec latency 32.439 ms 2024-03-26T01:58:09.884 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:09 smithi019 ceph-mon[24816]: Standby manager daemon smithi069.wpbewu restarted 2024-03-26T01:58:09.884 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:09 smithi019 ceph-mon[24816]: Standby manager daemon smithi069.wpbewu started 2024-03-26T01:58:09.885 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:09 smithi019 ceph-mon[24816]: from='mgr.? 172.21.15.69:0/312291043' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi069.wpbewu/crt"}]: dispatch 2024-03-26T01:58:09.885 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:09 smithi019 ceph-mon[24816]: from='mgr.? 172.21.15.69:0/312291043' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T01:58:09.885 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:09 smithi019 ceph-mon[24816]: from='mgr.? 172.21.15.69:0/312291043' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi069.wpbewu/key"}]: dispatch 2024-03-26T01:58:09.885 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:09 smithi019 ceph-mon[24816]: from='mgr.? 172.21.15.69:0/312291043' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T01:58:10.101 INFO:teuthology.orchestra.run.smithi019.stdout: 5 148367 12.08 MB/sec execute 163 sec latency 72.462 ms 2024-03-26T01:58:10.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:09 smithi069 ceph-mon[27071]: Standby manager daemon smithi069.wpbewu restarted 2024-03-26T01:58:10.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:09 smithi069 ceph-mon[27071]: Standby manager daemon smithi069.wpbewu started 2024-03-26T01:58:10.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:09 smithi069 ceph-mon[27071]: from='mgr.? 172.21.15.69:0/312291043' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi069.wpbewu/crt"}]: dispatch 2024-03-26T01:58:10.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:09 smithi069 ceph-mon[27071]: from='mgr.? 172.21.15.69:0/312291043' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T01:58:10.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:09 smithi069 ceph-mon[27071]: from='mgr.? 172.21.15.69:0/312291043' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi069.wpbewu/key"}]: dispatch 2024-03-26T01:58:10.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:09 smithi069 ceph-mon[27071]: from='mgr.? 172.21.15.69:0/312291043' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T01:58:10.939 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:10 smithi019 ceph-mon[24816]: pgmap v24: 97 pgs: 97 active+clean; 337 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 9.6 MiB/s wr, 956 op/s 2024-03-26T01:58:10.939 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:10 smithi019 ceph-mon[24816]: mgrmap e33: smithi019.ixuvdm(active, since 43s), standbys: smithi069.wpbewu 2024-03-26T01:58:10.939 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:10 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:10.939 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:10 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:10.939 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:10 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:10.940 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:10 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:10.940 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:10 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:10.940 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:10 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:58:10.940 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:10 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:10.940 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:10 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:10.940 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:10 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:10.940 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:10 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T01:58:10.940 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:10 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:10.940 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:10 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi019.ixuvdm"}]: dispatch 2024-03-26T01:58:10.940 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:10 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi019.ixuvdm"}]': finished 2024-03-26T01:58:10.940 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:10 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi069.wpbewu"}]: dispatch 2024-03-26T01:58:10.940 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:10 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi069.wpbewu"}]': finished 2024-03-26T01:58:10.940 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:10 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-26T01:58:11.101 INFO:teuthology.orchestra.run.smithi019.stdout: 5 148687 12.09 MB/sec execute 164 sec latency 32.769 ms 2024-03-26T01:58:11.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:10 smithi069 ceph-mon[27071]: pgmap v24: 97 pgs: 97 active+clean; 337 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 9.6 MiB/s wr, 956 op/s 2024-03-26T01:58:11.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:10 smithi069 ceph-mon[27071]: mgrmap e33: smithi019.ixuvdm(active, since 43s), standbys: smithi069.wpbewu 2024-03-26T01:58:11.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:10 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:11.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:10 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:11.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:10 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:11.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:10 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:11.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:10 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:11.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:10 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:58:11.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:10 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:11.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:10 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:11.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:10 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:11.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:10 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T01:58:11.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:10 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:11.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:10 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi019.ixuvdm"}]: dispatch 2024-03-26T01:58:11.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:10 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi019.ixuvdm"}]': finished 2024-03-26T01:58:11.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:10 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi069.wpbewu"}]: dispatch 2024-03-26T01:58:11.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:10 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi069.wpbewu"}]': finished 2024-03-26T01:58:11.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:10 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-26T01:58:11.954 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:11 smithi019 ceph-mon[24816]: Upgrade: Setting container_image for all mgr 2024-03-26T01:58:12.102 INFO:teuthology.orchestra.run.smithi019.stdout: 5 149085 12.09 MB/sec execute 165 sec latency 36.360 ms 2024-03-26T01:58:12.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:11 smithi069 ceph-mon[27071]: Upgrade: Setting container_image for all mgr 2024-03-26T01:58:12.968 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:12 smithi019 ceph-mon[24816]: pgmap v25: 97 pgs: 97 active+clean; 337 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 5.9 MiB/s wr, 598 op/s 2024-03-26T01:58:12.968 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:12 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:58:13.101 INFO:teuthology.orchestra.run.smithi019.stdout: 5 149602 12.10 MB/sec execute 166 sec latency 26.597 ms 2024-03-26T01:58:13.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:12 smithi069 ceph-mon[27071]: pgmap v25: 97 pgs: 97 active+clean; 337 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 5.9 MiB/s wr, 598 op/s 2024-03-26T01:58:13.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:12 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:58:14.102 INFO:teuthology.orchestra.run.smithi019.stdout: 5 149987 12.09 MB/sec execute 167 sec latency 31.632 ms 2024-03-26T01:58:14.480 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:14 smithi019 ceph-mon[24816]: pgmap v26: 97 pgs: 97 active+clean; 333 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 5.9 MiB/s wr, 600 op/s 2024-03-26T01:58:14.480 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:14 smithi019 ceph-mon[24816]: Upgrade: Updating mon.smithi019 2024-03-26T01:58:14.480 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:14 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:14.481 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:14 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:58:14.481 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:14 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T01:58:14.481 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:14 smithi019 ceph-mon[24816]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:14.481 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:14 smithi019 ceph-mon[24816]: Deploying daemon mon.smithi019 on smithi019 2024-03-26T01:58:14.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:14 smithi069 ceph-mon[27071]: pgmap v26: 97 pgs: 97 active+clean; 333 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 5.9 MiB/s wr, 600 op/s 2024-03-26T01:58:14.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:14 smithi069 ceph-mon[27071]: Upgrade: Updating mon.smithi019 2024-03-26T01:58:14.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:14 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:14.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:14 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:58:14.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:14 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T01:58:14.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:14 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:14.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:14 smithi069 ceph-mon[27071]: Deploying daemon mon.smithi019 on smithi019 2024-03-26T01:58:15.102 INFO:teuthology.orchestra.run.smithi019.stdout: 5 150495 12.05 MB/sec execute 168 sec latency 42.983 ms 2024-03-26T01:58:16.102 INFO:teuthology.orchestra.run.smithi019.stdout: 5 150936 12.06 MB/sec execute 169 sec latency 35.321 ms 2024-03-26T01:58:16.503 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:16 smithi019 ceph-mon[24816]: pgmap v27: 97 pgs: 97 active+clean; 340 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 8.4 MiB/s wr, 795 op/s 2024-03-26T01:58:16.503 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:16 smithi019 systemd[1]: Stopping Ceph mon.smithi019 for 692607c6-eb12-11ee-95d0-87774f69a715... 2024-03-26T01:58:16.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:16 smithi069 ceph-mon[27071]: pgmap v27: 97 pgs: 97 active+clean; 340 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 8.4 MiB/s wr, 795 op/s 2024-03-26T01:58:16.755 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:16 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi019[24812]: 2024-03-26T01:58:16.728+0000 7f8811b75700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi019 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T01:58:16.755 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:16 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi019[24812]: 2024-03-26T01:58:16.728+0000 7f8811b75700 -1 mon.smithi019@0(leader) e2 *** Got Signal Terminated *** 2024-03-26T01:58:17.006 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:16 smithi019 podman[66195]: 2024-03-26 01:58:16.821025608 +0000 UTC m=+0.231213153 container died b6908b497829a612ce36d3e1ff11366db26f10ad18ba6ea7804d82065d243974 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi019, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , distribution-scope=public, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, release=754, GIT_CLEAN=True, ceph=True, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0) 2024-03-26T01:58:17.102 INFO:teuthology.orchestra.run.smithi019.stdout: 5 151462 12.07 MB/sec execute 170 sec latency 39.611 ms 2024-03-26T01:58:17.508 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:17 smithi019 podman[66195]: 2024-03-26 01:58:17.401653283 +0000 UTC m=+0.811840824 container cleanup b6908b497829a612ce36d3e1ff11366db26f10ad18ba6ea7804d82065d243974 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi019, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, RELEASE=HEAD, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., io.openshift.expose-services=, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, GIT_BRANCH=HEAD, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, ceph=True) 2024-03-26T01:58:17.508 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:17 smithi019 bash[66195]: ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi019 2024-03-26T01:58:18.013 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:17 smithi019 podman[66216]: 2024-03-26 01:58:17.817449753 +0000 UTC m=+0.990744509 container remove b6908b497829a612ce36d3e1ff11366db26f10ad18ba6ea7804d82065d243974 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi019, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, version=8, vcs-type=git, GIT_CLEAN=True, distribution-scope=public, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container) 2024-03-26T01:58:18.102 INFO:teuthology.orchestra.run.smithi019.stdout: 5 152000 12.10 MB/sec execute 171 sec latency 52.759 ms 2024-03-26T01:58:18.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:18 smithi019 systemd[1]: ceph-692607c6-eb12-11ee-95d0-87774f69a715@mon.smithi019.service: Deactivated successfully. 2024-03-26T01:58:18.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:18 smithi019 systemd[1]: Stopped Ceph mon.smithi019 for 692607c6-eb12-11ee-95d0-87774f69a715. 2024-03-26T01:58:18.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:18 smithi019 systemd[1]: ceph-692607c6-eb12-11ee-95d0-87774f69a715@mon.smithi019.service: Consumed 16.883s CPU time. 2024-03-26T01:58:18.522 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:18 smithi019 systemd[1]: Starting Ceph mon.smithi019 for 692607c6-eb12-11ee-95d0-87774f69a715... 2024-03-26T01:58:18.776 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:18 smithi019 podman[66341]: 2024-03-26 01:58:18.53779753 +0000 UTC m=+0.029399970 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T01:58:18.776 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:18 smithi019 podman[66341]: 2024-03-26 01:58:18.672446424 +0000 UTC m=+0.164048852 container create c18934705774976b47d22d796fcba2731576c23adc1015d6071f9e880e27e628 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi019, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-26T01:58:19.102 INFO:teuthology.orchestra.run.smithi019.stdout: 5 152379 12.11 MB/sec execute 172 sec latency 31.391 ms 2024-03-26T01:58:19.278 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 podman[66341]: 2024-03-26 01:58:19.213406099 +0000 UTC m=+0.705008534 container init c18934705774976b47d22d796fcba2731576c23adc1015d6071f9e880e27e628 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi019, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T01:58:19.278 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 podman[66341]: 2024-03-26 01:58:19.221523731 +0000 UTC m=+0.713126161 container start c18934705774976b47d22d796fcba2731576c23adc1015d6071f9e880e27e628 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi019, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-26T01:58:19.278 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T01:58:19.278 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev), process ceph-mon, pid 2 2024-03-26T01:58:19.278 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: pidfile_write: ignore empty --pid-file 2024-03-26T01:58:19.278 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: load: jerasure load: lrc 2024-03-26T01:58:19.278 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: RocksDB version: 7.9.2 2024-03-26T01:58:19.278 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Git sha 0 2024-03-26T01:58:19.278 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Compile date 2024-03-25 16:14:20 2024-03-26T01:58:19.278 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: DB SUMMARY 2024-03-26T01:58:19.278 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: DB Session ID: 9BEYJDOMYB7CZAKIZAGZ 2024-03-26T01:58:19.279 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: CURRENT file: CURRENT 2024-03-26T01:58:19.279 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T01:58:19.279 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: MANIFEST file: MANIFEST-000015 size: 978 Bytes 2024-03-26T01:58:19.279 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi019/store.db dir, Total Num: 1, files: 000030.sst 2024-03-26T01:58:19.279 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi019/store.db: 000028.log size: 2901840 ; 2024-03-26T01:58:19.279 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.error_if_exists: 0 2024-03-26T01:58:19.279 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.create_if_missing: 0 2024-03-26T01:58:19.279 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.paranoid_checks: 1 2024-03-26T01:58:19.279 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T01:58:19.279 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T01:58:19.279 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T01:58:19.279 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.env: 0x564acb219ae0 2024-03-26T01:58:19.279 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T01:58:19.280 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.info_log: 0x564acd2017c0 2024-03-26T01:58:19.280 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T01:58:19.280 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.statistics: (nil) 2024-03-26T01:58:19.280 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.use_fsync: 0 2024-03-26T01:58:19.280 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.max_log_file_size: 0 2024-03-26T01:58:19.280 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T01:58:19.280 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T01:58:19.280 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T01:58:19.280 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T01:58:19.280 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.allow_fallocate: 1 2024-03-26T01:58:19.280 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T01:58:19.280 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T01:58:19.280 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.use_direct_reads: 0 2024-03-26T01:58:19.280 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T01:58:19.280 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T01:58:19.281 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.db_log_dir: 2024-03-26T01:58:19.281 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.wal_dir: 2024-03-26T01:58:19.281 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T01:58:19.281 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T01:58:19.281 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T01:58:19.281 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T01:58:19.281 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T01:58:19.281 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T01:58:19.281 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T01:58:19.281 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T01:58:19.281 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.write_buffer_manager: 0x564acd1d41e0 2024-03-26T01:58:19.281 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T01:58:19.281 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T01:58:19.282 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T01:58:19.282 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T01:58:19.282 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T01:58:19.282 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T01:58:19.282 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T01:58:19.282 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T01:58:19.282 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.unordered_write: 0 2024-03-26T01:58:19.282 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T01:58:19.282 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T01:58:19.282 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T01:58:19.282 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T01:58:19.282 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.row_cache: None 2024-03-26T01:58:19.282 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.wal_filter: None 2024-03-26T01:58:19.284 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T01:58:19.284 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T01:58:19.284 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.two_write_queues: 0 2024-03-26T01:58:19.284 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T01:58:19.284 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.wal_compression: 0 2024-03-26T01:58:19.284 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.atomic_flush: 0 2024-03-26T01:58:19.284 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T01:58:19.284 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T01:58:19.284 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T01:58:19.284 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.log_readahead_size: 0 2024-03-26T01:58:19.284 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T01:58:19.284 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T01:58:19.284 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T01:58:19.285 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T01:58:19.285 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T01:58:19.285 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T01:58:19.285 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T01:58:19.285 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.max_background_jobs: 2 2024-03-26T01:58:19.285 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.max_background_compactions: -1 2024-03-26T01:58:19.285 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.max_subcompactions: 1 2024-03-26T01:58:19.285 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T01:58:19.285 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T01:58:19.285 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T01:58:19.285 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T01:58:19.285 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T01:58:19.285 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T01:58:19.285 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T01:58:19.285 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T01:58:19.286 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.max_open_files: -1 2024-03-26T01:58:19.286 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T01:58:19.286 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T01:58:19.286 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T01:58:19.286 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T01:58:19.286 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.max_background_flushes: -1 2024-03-26T01:58:19.286 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Compression algorithms supported: 2024-03-26T01:58:19.286 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T01:58:19.286 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: kZSTD supported: 0 2024-03-26T01:58:19.286 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: kXpressCompression supported: 0 2024-03-26T01:58:19.286 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T01:58:19.286 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: kZlibCompression supported: 1 2024-03-26T01:58:19.286 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: kSnappyCompression supported: 1 2024-03-26T01:58:19.287 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: kLZ4Compression supported: 1 2024-03-26T01:58:19.287 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: kBZip2Compression supported: 0 2024-03-26T01:58:19.287 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T01:58:19.287 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T01:58:19.287 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi019/store.db/MANIFEST-000015 2024-03-26T01:58:19.287 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T01:58:19.287 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T01:58:19.287 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.merge_operator: 2024-03-26T01:58:19.287 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.compaction_filter: None 2024-03-26T01:58:19.287 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T01:58:19.287 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T01:58:19.287 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T01:58:19.287 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T01:58:19.287 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564acd2018e0) 2024-03-26T01:58:19.287 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: cache_index_and_filter_blocks: 1 2024-03-26T01:58:19.288 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T01:58:19.288 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T01:58:19.288 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: pin_top_level_index_and_filter: 1 2024-03-26T01:58:19.288 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: index_type: 0 2024-03-26T01:58:19.288 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: data_block_index_type: 0 2024-03-26T01:58:19.288 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: index_shortening: 1 2024-03-26T01:58:19.288 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T01:58:19.288 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: checksum: 4 2024-03-26T01:58:19.288 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: no_block_cache: 0 2024-03-26T01:58:19.288 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: block_cache: 0x564acd205090 2024-03-26T01:58:19.288 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: block_cache_name: BinnedLRUCache 2024-03-26T01:58:19.288 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: block_cache_options: 2024-03-26T01:58:19.288 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: capacity : 536870912 2024-03-26T01:58:19.288 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: num_shard_bits : 4 2024-03-26T01:58:19.288 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: strict_capacity_limit : 0 2024-03-26T01:58:19.288 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: high_pri_pool_ratio: 0.000 2024-03-26T01:58:19.289 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: block_cache_compressed: (nil) 2024-03-26T01:58:19.289 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: persistent_cache: (nil) 2024-03-26T01:58:19.289 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: block_size: 4096 2024-03-26T01:58:19.289 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: block_size_deviation: 10 2024-03-26T01:58:19.289 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: block_restart_interval: 16 2024-03-26T01:58:19.289 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: index_block_restart_interval: 1 2024-03-26T01:58:19.289 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: metadata_block_size: 4096 2024-03-26T01:58:19.289 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: partition_filters: 0 2024-03-26T01:58:19.289 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: use_delta_encoding: 1 2024-03-26T01:58:19.289 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: filter_policy: bloomfilter 2024-03-26T01:58:19.289 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: whole_key_filtering: 1 2024-03-26T01:58:19.289 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: verify_compression: 0 2024-03-26T01:58:19.289 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: read_amp_bytes_per_bit: 0 2024-03-26T01:58:19.289 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: format_version: 5 2024-03-26T01:58:19.289 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: enable_index_compression: 1 2024-03-26T01:58:19.290 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: block_align: 0 2024-03-26T01:58:19.290 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: max_auto_readahead_size: 262144 2024-03-26T01:58:19.290 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: prepopulate_block_cache: 0 2024-03-26T01:58:19.290 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: initial_auto_readahead_size: 8192 2024-03-26T01:58:19.290 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T01:58:19.290 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T01:58:19.290 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T01:58:19.290 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.compression: NoCompression 2024-03-26T01:58:19.290 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T01:58:19.290 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T01:58:19.290 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T01:58:19.290 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.num_levels: 7 2024-03-26T01:58:19.290 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T01:58:19.290 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T01:58:19.291 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T01:58:19.291 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T01:58:19.291 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T01:58:19.291 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T01:58:19.291 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T01:58:19.291 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T01:58:19.291 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T01:58:19.291 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T01:58:19.291 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T01:58:19.291 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T01:58:19.291 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T01:58:19.291 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T01:58:19.291 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T01:58:19.291 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T01:58:19.291 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T01:58:19.291 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T01:58:19.292 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T01:58:19.292 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T01:58:19.292 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T01:58:19.292 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T01:58:19.292 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T01:58:19.292 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T01:58:19.292 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T01:58:19.292 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T01:58:19.292 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T01:58:19.292 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T01:58:19.292 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T01:58:19.292 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T01:58:19.292 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T01:58:19.292 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T01:58:19.292 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T01:58:19.293 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T01:58:19.293 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T01:58:19.293 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T01:58:19.293 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T01:58:19.293 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T01:58:19.293 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T01:58:19.293 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T01:58:19.293 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T01:58:19.293 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T01:58:19.293 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T01:58:19.293 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T01:58:19.293 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T01:58:19.293 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T01:58:19.293 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T01:58:19.293 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T01:58:19.294 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T01:58:19.294 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T01:58:19.294 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T01:58:19.294 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T01:58:19.294 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T01:58:19.294 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.table_properties_collectors: 2024-03-26T01:58:19.294 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.inplace_update_support: 0 2024-03-26T01:58:19.294 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T01:58:19.294 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T01:58:19.295 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T01:58:19.295 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T01:58:19.295 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.bloom_locality: 0 2024-03-26T01:58:19.295 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.max_successive_merges: 0 2024-03-26T01:58:19.295 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T01:58:19.295 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T01:58:19.295 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T01:58:19.295 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T01:58:19.295 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.ttl: 2592000 2024-03-26T01:58:19.295 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T01:58:19.295 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T01:58:19.295 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T01:58:19.295 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.enable_blob_files: false 2024-03-26T01:58:19.295 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.min_blob_size: 0 2024-03-26T01:58:19.296 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T01:58:19.296 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T01:58:19.296 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T01:58:19.296 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T01:58:19.296 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T01:58:19.296 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T01:58:19.296 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T01:58:19.296 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T01:58:19.296 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-03-26T01:58:19.296 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-26T01:58:19.296 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi019/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 11819, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-26T01:58:19.296 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-03-26T01:58:19.296 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 640503e3-0bd5-4503-a0b2-1c65f53f3832 2024-03-26T01:58:19.296 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418299261335, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-03-26T01:58:19.296 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-03-26T01:58:19.297 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418299272804, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 2666711, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11820, "largest_seqno": 13015, "table_properties": {"data_size": 2660864, "index_size": 3548, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1285, "raw_key_size": 11711, "raw_average_key_size": 22, "raw_value_size": 2649777, "raw_average_value_size": 5185, "num_data_blocks": 161, "num_entries": 511, "num_filter_entries": 511, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711418299, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "640503e3-0bd5-4503-a0b2-1c65f53f3832", "db_session_id": "9BEYJDOMYB7CZAKIZAGZ", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-03-26T01:58:19.297 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418299272953, "job": 1, "event": "recovery_finished"} 2024-03-26T01:58:19.297 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-03-26T01:58:19.297 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-26T01:58:19.297 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi019/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T01:58:19.297 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x564acd2f4000 2024-03-26T01:58:19.297 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: DB pointer 0x564acd2e0000 2024-03-26T01:58:19.297 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T01:58:19.297 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T01:58:19.297 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: ** DB Stats ** 2024-03-26T01:58:19.297 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T01:58:19.297 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T01:58:19.297 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T01:58:19.297 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T01:58:19.298 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T01:58:19.298 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T01:58:19.298 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T01:58:19.298 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: 2024-03-26T01:58:19.298 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: ** Compaction Stats [default] ** 2024-03-26T01:58:19.298 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T01:58:19.298 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T01:58:19.298 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: L0 1/0 2.54 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 457.9 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-03-26T01:58:19.298 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: L6 1/0 6.25 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-26T01:58:19.298 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Sum 2/0 8.79 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 457.9 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-03-26T01:58:19.298 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 457.9 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-03-26T01:58:19.298 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: 2024-03-26T01:58:19.298 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: ** Compaction Stats [default] ** 2024-03-26T01:58:19.298 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T01:58:19.299 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T01:58:19.299 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 457.9 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-03-26T01:58:19.299 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: 2024-03-26T01:58:19.299 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T01:58:19.299 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: 2024-03-26T01:58:19.299 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T01:58:19.299 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-03-26T01:58:19.299 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T01:58:19.299 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T01:58:19.299 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T01:58:19.299 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T01:58:19.299 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Cumulative compaction: 0.00 GB write, 149.21 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T01:58:19.299 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Interval compaction: 0.00 GB write, 149.21 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T01:58:19.299 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T01:58:19.300 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Block cache BinnedLRUCache@0x564acd205090#2 capacity: 512.00 MB usage: 5.48 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-26T01:58:19.300 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.41 KB,0.000268221%) IndexBlock(1,4.08 KB,0.000777841%) Misc(1,0.00 KB,0%) 2024-03-26T01:58:19.300 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: 2024-03-26T01:58:19.300 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T01:58:19.300 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: starting mon.smithi019 rank 0 at public addrs [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] at bind addrs [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi019 fsid 692607c6-eb12-11ee-95d0-87774f69a715 2024-03-26T01:58:19.300 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: mon.smithi019@-1(???) e2 preinit fsid 692607c6-eb12-11ee-95d0-87774f69a715 2024-03-26T01:58:19.300 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: mon.smithi019@-1(???).mds e7 new map 2024-03-26T01:58:19.300 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: mon.smithi019@-1(???).mds e7 print_map 2024-03-26T01:58:19.300 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: e7 2024-03-26T01:58:19.300 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T01:58:19.300 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T01:58:19.300 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: legacy client fscid: 1 2024-03-26T01:58:19.300 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: 2024-03-26T01:58:19.300 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Filesystem 'foofs' (1) 2024-03-26T01:58:19.300 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: fs_name foofs 2024-03-26T01:58:19.301 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: epoch 5 2024-03-26T01:58:19.301 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-26T01:58:19.301 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: created 2024-03-26T01:52:48.813757+0000 2024-03-26T01:58:19.301 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: modified 2024-03-26T01:52:54.832475+0000 2024-03-26T01:58:19.301 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: tableserver 0 2024-03-26T01:58:19.301 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: root 0 2024-03-26T01:58:19.301 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: session_timeout 60 2024-03-26T01:58:19.301 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: session_autoclose 300 2024-03-26T01:58:19.301 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: max_file_size 1099511627776 2024-03-26T01:58:19.301 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: max_xattr_size 65536 2024-03-26T01:58:19.301 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: required_client_features {} 2024-03-26T01:58:19.301 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: last_failure 0 2024-03-26T01:58:19.301 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: last_failure_osd_epoch 0 2024-03-26T01:58:19.301 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T01:58:19.301 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: max_mds 1 2024-03-26T01:58:19.302 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: in 0 2024-03-26T01:58:19.302 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: up {0=14478} 2024-03-26T01:58:19.302 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: failed 2024-03-26T01:58:19.302 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: damaged 2024-03-26T01:58:19.302 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: stopped 2024-03-26T01:58:19.302 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: data_pools [3] 2024-03-26T01:58:19.302 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: metadata_pool 2 2024-03-26T01:58:19.302 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: inline_data disabled 2024-03-26T01:58:19.302 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: balancer 2024-03-26T01:58:19.302 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: bal_rank_mask -1 2024-03-26T01:58:19.302 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: standby_count_wanted 1 2024-03-26T01:58:19.302 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: qdb_cluster leader: 0 members: 2024-03-26T01:58:19.302 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: [mds.foofs.smithi019.lmryom{0:14478} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.19:6834/3267072550,v1:172.21.15.19:6835/3267072550] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T01:58:19.302 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: 2024-03-26T01:58:19.302 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: 2024-03-26T01:58:19.303 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: Standby daemons: 2024-03-26T01:58:19.303 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: 2024-03-26T01:58:19.303 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout: [mds.foofs.smithi069.spgwgo{-1:24309} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.69:6832/641242727,v1:172.21.15.69:6833/641242727] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T01:58:19.303 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: mon.smithi019@-1(???).osd e72 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-26T01:58:19.303 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: mon.smithi019@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-03-26T01:58:19.303 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: mon.smithi019@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-03-26T01:58:19.303 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: mon.smithi019@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-03-26T01:58:19.303 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: mon.smithi019@-1(???).paxosservice(auth 1..29) refresh upgraded, format 0 -> 3 2024-03-26T01:58:19.531 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 bash[66341]: c18934705774976b47d22d796fcba2731576c23adc1015d6071f9e880e27e628 2024-03-26T01:58:19.531 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 systemd[1]: Started Ceph mon.smithi019 for 692607c6-eb12-11ee-95d0-87774f69a715. 2024-03-26T01:58:20.035 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi019"}]: dispatch 2024-03-26T01:58:20.035 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi069"}]: dispatch 2024-03-26T01:58:20.035 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: pgmap v28: 97 pgs: 97 active+clean; 340 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.5 MiB/s wr, 526 op/s 2024-03-26T01:58:20.035 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: mon.smithi019 calling monitor election 2024-03-26T01:58:20.036 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: mon.smithi019 is new leader, mons smithi019,smithi069 in quorum (ranks 0,1) 2024-03-26T01:58:20.036 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: monmap e2: 2 mons at {smithi019=[v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0],smithi069=[v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T01:58:20.036 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: fsmap foofs:1 {0=foofs.smithi019.lmryom=up:active} 1 up:standby 2024-03-26T01:58:20.036 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T01:58:20.036 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: mgrmap e33: smithi019.ixuvdm(active, since 53s), standbys: smithi069.wpbewu 2024-03-26T01:58:20.036 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: overall HEALTH_OK 2024-03-26T01:58:20.036 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: from='mgr.14758 ' entity='' 2024-03-26T01:58:20.036 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:19 smithi019 ceph-mon[66359]: mgrmap e34: smithi019.ixuvdm(active, since 53s), standbys: smithi069.wpbewu 2024-03-26T01:58:20.102 INFO:teuthology.orchestra.run.smithi019.stdout: 5 152789 12.10 MB/sec execute 173 sec latency 39.574 ms 2024-03-26T01:58:20.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:19 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi019"}]: dispatch 2024-03-26T01:58:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:19 smithi069 ceph-mon[27071]: from='mgr.14758 172.21.15.19:0/175505519' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi069"}]: dispatch 2024-03-26T01:58:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:19 smithi069 ceph-mon[27071]: pgmap v28: 97 pgs: 97 active+clean; 340 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.5 MiB/s wr, 526 op/s 2024-03-26T01:58:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:19 smithi069 ceph-mon[27071]: mon.smithi019 calling monitor election 2024-03-26T01:58:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:19 smithi069 ceph-mon[27071]: mon.smithi019 is new leader, mons smithi019,smithi069 in quorum (ranks 0,1) 2024-03-26T01:58:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:19 smithi069 ceph-mon[27071]: monmap e2: 2 mons at {smithi019=[v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0],smithi069=[v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T01:58:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:19 smithi069 ceph-mon[27071]: fsmap foofs:1 {0=foofs.smithi019.lmryom=up:active} 1 up:standby 2024-03-26T01:58:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:19 smithi069 ceph-mon[27071]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T01:58:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:19 smithi069 ceph-mon[27071]: mgrmap e33: smithi019.ixuvdm(active, since 53s), standbys: smithi069.wpbewu 2024-03-26T01:58:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:19 smithi069 ceph-mon[27071]: overall HEALTH_OK 2024-03-26T01:58:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:19 smithi069 ceph-mon[27071]: from='mgr.14758 ' entity='' 2024-03-26T01:58:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:19 smithi069 ceph-mon[27071]: mgrmap e34: smithi019.ixuvdm(active, since 53s), standbys: smithi069.wpbewu 2024-03-26T01:58:21.102 INFO:teuthology.orchestra.run.smithi019.stdout: 5 153074 12.09 MB/sec execute 174 sec latency 46.411 ms 2024-03-26T01:58:22.075 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:21 smithi019 ceph-mon[66359]: mgrmap e35: smithi019.ixuvdm(active, since 54s), standbys: smithi069.wpbewu 2024-03-26T01:58:22.102 INFO:teuthology.orchestra.run.smithi019.stdout: 5 153362 12.06 MB/sec execute 175 sec latency 43.230 ms 2024-03-26T01:58:22.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:21 smithi069 ceph-mon[27071]: mgrmap e35: smithi019.ixuvdm(active, since 54s), standbys: smithi069.wpbewu 2024-03-26T01:58:23.102 INFO:teuthology.orchestra.run.smithi019.stdout: 5 154083 12.06 MB/sec execute 176 sec latency 32.837 ms 2024-03-26T01:58:24.102 INFO:teuthology.orchestra.run.smithi019.stdout: 5 154682 12.09 MB/sec execute 177 sec latency 27.025 ms 2024-03-26T01:58:25.102 INFO:teuthology.orchestra.run.smithi019.stdout: 5 155194 12.08 MB/sec execute 178 sec latency 58.205 ms 2024-03-26T01:58:26.102 INFO:teuthology.orchestra.run.smithi019.stdout: 5 155610 12.11 MB/sec execute 179 sec latency 40.048 ms 2024-03-26T01:58:27.102 INFO:teuthology.orchestra.run.smithi019.stdout: 5 156023 12.13 MB/sec execute 180 sec latency 50.689 ms 2024-03-26T01:58:28.102 INFO:teuthology.orchestra.run.smithi019.stdout: 5 156397 12.12 MB/sec execute 181 sec latency 42.696 ms 2024-03-26T01:58:28.379 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:28 smithi019 ceph-mon[66359]: from='mgr.? 172.21.15.69:0/1347178413' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi069.wpbewu/crt"}]: dispatch 2024-03-26T01:58:28.379 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:28 smithi019 ceph-mon[66359]: from='mgr.? 172.21.15.69:0/1347178413' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T01:58:28.379 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:28 smithi019 ceph-mon[66359]: Standby manager daemon smithi069.wpbewu restarted 2024-03-26T01:58:28.380 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:28 smithi019 ceph-mon[66359]: Standby manager daemon smithi069.wpbewu started 2024-03-26T01:58:28.380 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:28 smithi019 ceph-mon[66359]: from='mgr.? 172.21.15.69:0/1347178413' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi069.wpbewu/key"}]: dispatch 2024-03-26T01:58:28.380 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:28 smithi019 ceph-mon[66359]: from='mgr.? 172.21.15.69:0/1347178413' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T01:58:28.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:28 smithi069 ceph-mon[27071]: from='mgr.? 172.21.15.69:0/1347178413' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi069.wpbewu/crt"}]: dispatch 2024-03-26T01:58:28.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:28 smithi069 ceph-mon[27071]: from='mgr.? 172.21.15.69:0/1347178413' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T01:58:28.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:28 smithi069 ceph-mon[27071]: Standby manager daemon smithi069.wpbewu restarted 2024-03-26T01:58:28.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:28 smithi069 ceph-mon[27071]: Standby manager daemon smithi069.wpbewu started 2024-03-26T01:58:28.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:28 smithi069 ceph-mon[27071]: from='mgr.? 172.21.15.69:0/1347178413' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi069.wpbewu/key"}]: dispatch 2024-03-26T01:58:28.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:28 smithi069 ceph-mon[27071]: from='mgr.? 172.21.15.69:0/1347178413' entity='mgr.smithi069.wpbewu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T01:58:29.102 INFO:teuthology.orchestra.run.smithi019.stdout: 5 156686 12.09 MB/sec execute 182 sec latency 32.550 ms 2024-03-26T01:58:29.398 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:29 smithi019 ceph-mon[66359]: mgrmap e36: smithi019.ixuvdm(active, since 62s), standbys: smithi069.wpbewu 2024-03-26T01:58:29.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:29 smithi069 ceph-mon[27071]: mgrmap e36: smithi019.ixuvdm(active, since 62s), standbys: smithi069.wpbewu 2024-03-26T01:58:30.103 INFO:teuthology.orchestra.run.smithi019.stdout: 5 157152 12.08 MB/sec execute 183 sec latency 48.693 ms 2024-03-26T01:58:30.728 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:30 smithi019 ceph-mon[66359]: Active manager daemon smithi019.ixuvdm restarted 2024-03-26T01:58:30.728 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:30 smithi019 ceph-mon[66359]: Activating manager daemon smithi019.ixuvdm 2024-03-26T01:58:30.728 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:30 smithi019 ceph-mon[66359]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T01:58:30.728 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:30 smithi019 ceph-mon[66359]: mgrmap e37: smithi019.ixuvdm(active, starting, since 0.00823603s), standbys: smithi069.wpbewu 2024-03-26T01:58:30.728 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi019"}]: dispatch 2024-03-26T01:58:30.728 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi069"}]: dispatch 2024-03-26T01:58:30.728 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi019.lmryom"}]: dispatch 2024-03-26T01:58:30.728 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi069.spgwgo"}]: dispatch 2024-03-26T01:58:30.728 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mgr metadata", "who": "smithi019.ixuvdm", "id": "smithi019.ixuvdm"}]: dispatch 2024-03-26T01:58:30.728 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mgr metadata", "who": "smithi069.wpbewu", "id": "smithi069.wpbewu"}]: dispatch 2024-03-26T01:58:30.728 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:58:30.728 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:58:30.729 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:30.729 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:58:30.729 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T01:58:30.729 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T01:58:30.729 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T01:58:30.729 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T01:58:30.729 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T01:58:30.729 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T01:58:30.729 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T01:58:30.729 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:30 smithi019 ceph-mon[66359]: Manager daemon smithi019.ixuvdm is now available 2024-03-26T01:58:30.878 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:30 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi069[27054]: 2024-03-26T01:58:30.625+0000 7f93ddb74700 -1 mon.smithi069@1(peon).osd e73 update_from_paxos full map CRC mismatch, resetting to canonical 2024-03-26T01:58:30.878 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:30 smithi069 ceph-mon[27071]: Active manager daemon smithi019.ixuvdm restarted 2024-03-26T01:58:30.879 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:30 smithi069 ceph-mon[27071]: Activating manager daemon smithi019.ixuvdm 2024-03-26T01:58:30.879 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:30 smithi069 ceph-mon[27071]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T01:58:30.879 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:30 smithi069 ceph-mon[27071]: mgrmap e37: smithi019.ixuvdm(active, starting, since 0.00823603s), standbys: smithi069.wpbewu 2024-03-26T01:58:30.879 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:30 smithi069 ceph-mon[27071]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi019"}]: dispatch 2024-03-26T01:58:30.879 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:30 smithi069 ceph-mon[27071]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi069"}]: dispatch 2024-03-26T01:58:30.879 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:30 smithi069 ceph-mon[27071]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi019.lmryom"}]: dispatch 2024-03-26T01:58:30.879 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:30 smithi069 ceph-mon[27071]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi069.spgwgo"}]: dispatch 2024-03-26T01:58:30.879 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:30 smithi069 ceph-mon[27071]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mgr metadata", "who": "smithi019.ixuvdm", "id": "smithi019.ixuvdm"}]: dispatch 2024-03-26T01:58:30.879 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:30 smithi069 ceph-mon[27071]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mgr metadata", "who": "smithi069.wpbewu", "id": "smithi069.wpbewu"}]: dispatch 2024-03-26T01:58:30.879 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:30 smithi069 ceph-mon[27071]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:58:30.879 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:30 smithi069 ceph-mon[27071]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:58:30.879 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:30 smithi069 ceph-mon[27071]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:30.879 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:30 smithi069 ceph-mon[27071]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:58:30.879 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:30 smithi069 ceph-mon[27071]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T01:58:30.880 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:30 smithi069 ceph-mon[27071]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T01:58:30.880 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:30 smithi069 ceph-mon[27071]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T01:58:30.880 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:30 smithi069 ceph-mon[27071]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T01:58:30.880 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:30 smithi069 ceph-mon[27071]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T01:58:30.880 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:30 smithi069 ceph-mon[27071]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T01:58:30.880 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:30 smithi069 ceph-mon[27071]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T01:58:30.880 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:30 smithi069 ceph-mon[27071]: Manager daemon smithi019.ixuvdm is now available 2024-03-26T01:58:31.103 INFO:teuthology.orchestra.run.smithi019.stdout: 5 157695 12.06 MB/sec execute 184 sec latency 38.238 ms 2024-03-26T01:58:31.665 INFO:teuthology.orchestra.run.smithi019.stdout:true 2024-03-26T01:58:31.751 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:31 smithi019 ceph-mon[66359]: failed to encode map e73 with expected crc 2024-03-26T01:58:31.752 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:31 smithi019 ceph-mon[66359]: failed to encode map e73 with expected crc 2024-03-26T01:58:31.752 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:31 smithi019 ceph-mon[66359]: failed to encode map e73 with expected crc 2024-03-26T01:58:31.752 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:31 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:31.752 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:31 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:31.752 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:31 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:58:31.752 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:31 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:58:31.752 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:31 smithi019 ceph-mon[66359]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:58:31.752 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:31 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/trash_purge_schedule"}]: dispatch 2024-03-26T01:58:31.752 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:31 smithi019 ceph-mon[66359]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/trash_purge_schedule"}]: dispatch 2024-03-26T01:58:31.752 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:31 smithi019 ceph-mon[66359]: mgrmap e38: smithi019.ixuvdm(active, since 1.01128s), standbys: smithi069.wpbewu 2024-03-26T01:58:32.103 INFO:teuthology.orchestra.run.smithi019.stdout: 5 158002 12.02 MB/sec execute 185 sec latency 63.287 ms 2024-03-26T01:58:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:31 smithi069 ceph-mon[27071]: failed to encode map e73 with expected crc 2024-03-26T01:58:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:31 smithi069 ceph-mon[27071]: failed to encode map e73 with expected crc 2024-03-26T01:58:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:31 smithi069 ceph-mon[27071]: failed to encode map e73 with expected crc 2024-03-26T01:58:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:31 smithi069 ceph-mon[27071]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:31 smithi069 ceph-mon[27071]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:31 smithi069 ceph-mon[27071]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:58:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:31 smithi069 ceph-mon[27071]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:58:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:31 smithi069 ceph-mon[27071]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:58:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:31 smithi069 ceph-mon[27071]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/trash_purge_schedule"}]: dispatch 2024-03-26T01:58:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:31 smithi069 ceph-mon[27071]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi019.ixuvdm/trash_purge_schedule"}]: dispatch 2024-03-26T01:58:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:31 smithi069 ceph-mon[27071]: mgrmap e38: smithi019.ixuvdm(active, since 1.01128s), standbys: smithi069.wpbewu 2024-03-26T01:58:32.752 INFO:teuthology.orchestra.run.smithi019.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T01:58:32.752 INFO:teuthology.orchestra.run.smithi019.stdout:alertmanager.smithi019 smithi019 *:9093,9094 running (2m) 22s ago 11m 16.2M - 0.23.0 ba2b418f427c a6630841c2b2 2024-03-26T01:58:32.752 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi019 smithi019 running (11m) 22s ago 11m 6405k - 17.2.0 e1d6a67b021e 713db693d5c1 2024-03-26T01:58:32.752 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi069 smithi069 running (11m) 27s ago 11m 6405k - 17.2.0 e1d6a67b021e a87d27e2b74b 2024-03-26T01:58:32.752 INFO:teuthology.orchestra.run.smithi019.stdout:grafana.smithi019 smithi019 *:3000 running (2m) 22s ago 11m 47.4M - 8.3.5 dad864ee21e9 a97dea3b5fec 2024-03-26T01:58:32.753 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi019.lmryom smithi019 running (5m) 22s ago 5m 37.6M - 17.2.0 e1d6a67b021e 91a7c2942715 2024-03-26T01:58:32.753 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi069.spgwgo smithi069 running (5m) 27s ago 5m 13.7M - 17.2.0 e1d6a67b021e 94d96338dbe1 2024-03-26T01:58:32.753 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi019.ixuvdm smithi019 *:8443,9283,8765 running (85s) 22s ago 12m 496M - 19.0.0-1775-g5e70124b 5f21c9264136 4460b22c7a56 2024-03-26T01:58:32.753 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi069.wpbewu smithi069 *:8443,9283,8765 running (32s) 27s ago 10m 65.4M - 19.0.0-1775-g5e70124b 5f21c9264136 380fd33842a8 2024-03-26T01:58:32.753 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi019 smithi019 running (13m) 22s ago 13m 57.1M 2048M 17.2.0 e1d6a67b021e b6908b497829 2024-03-26T01:58:32.753 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi069 smithi069 running (10m) 27s ago 10m 45.9M 2048M 17.2.0 e1d6a67b021e 2d51b7b7bb32 2024-03-26T01:58:32.753 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.0.0.smithi019.qbizxo smithi019 *:2049 running (5m) 22s ago 5m 283M - 4.0 e1d6a67b021e cd6628a9075d 2024-03-26T01:58:32.753 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.1.0.smithi069.uawdhc smithi069 *:2049 running (5m) 27s ago 5m 85.1M - 4.0 e1d6a67b021e 6ee3695e7a10 2024-03-26T01:58:32.753 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi019 smithi019 *:9100 running (2m) 22s ago 11m 9705k - 1.3.1 1dbe0e931976 0d1d184bec63 2024-03-26T01:58:32.753 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi069 smithi069 *:9100 running (96s) 27s ago 10m 12.5M - 1.3.1 1dbe0e931976 c49a006f94e7 2024-03-26T01:58:32.753 INFO:teuthology.orchestra.run.smithi019.stdout:osd.0 smithi019 running (10m) 22s ago 10m 500M 1937M 17.2.0 e1d6a67b021e c1a797230ab6 2024-03-26T01:58:32.753 INFO:teuthology.orchestra.run.smithi019.stdout:osd.1 smithi019 running (9m) 22s ago 9m 431M 1937M 17.2.0 e1d6a67b021e 31d3489c7188 2024-03-26T01:58:32.753 INFO:teuthology.orchestra.run.smithi019.stdout:osd.2 smithi019 running (9m) 22s ago 9m 312M 1937M 17.2.0 e1d6a67b021e f549115e7685 2024-03-26T01:58:32.753 INFO:teuthology.orchestra.run.smithi019.stdout:osd.3 smithi019 running (8m) 22s ago 8m 460M 1937M 17.2.0 e1d6a67b021e 154af68d03e1 2024-03-26T01:58:32.754 INFO:teuthology.orchestra.run.smithi019.stdout:osd.4 smithi069 running (8m) 27s ago 8m 478M 2704M 17.2.0 e1d6a67b021e 4fd5e853f8bf 2024-03-26T01:58:32.754 INFO:teuthology.orchestra.run.smithi019.stdout:osd.5 smithi069 running (7m) 27s ago 7m 554M 2704M 17.2.0 e1d6a67b021e 932e401d87c9 2024-03-26T01:58:32.754 INFO:teuthology.orchestra.run.smithi019.stdout:osd.6 smithi069 running (7m) 27s ago 7m 464M 2704M 17.2.0 e1d6a67b021e b123f75154c8 2024-03-26T01:58:32.754 INFO:teuthology.orchestra.run.smithi019.stdout:osd.7 smithi069 running (7m) 27s ago 7m 373M 2704M 17.2.0 e1d6a67b021e d175fbc9ff8a 2024-03-26T01:58:32.754 INFO:teuthology.orchestra.run.smithi019.stdout:prometheus.smithi019 smithi019 *:9095 running (39s) 22s ago 10m 51.3M - 2.33.4 514e6a882f6e 29f89db72a07 2024-03-26T01:58:32.760 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:32 smithi019 ceph-mon[66359]: [26/Mar/2024:01:58:31] ENGINE Bus STARTING 2024-03-26T01:58:32.760 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:32 smithi019 ceph-mon[66359]: [26/Mar/2024:01:58:31] ENGINE Serving on https://172.21.15.19:7150 2024-03-26T01:58:32.760 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:32 smithi019 ceph-mon[66359]: from='client.34108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:58:32.760 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:32 smithi019 ceph-mon[66359]: pgmap v3: 97 pgs: 97 active+clean; 331 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-03-26T01:58:32.760 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:32 smithi019 ceph-mon[66359]: [26/Mar/2024:01:58:31] ENGINE Serving on http://172.21.15.19:8765 2024-03-26T01:58:32.760 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:32 smithi019 ceph-mon[66359]: [26/Mar/2024:01:58:31] ENGINE Bus STARTED 2024-03-26T01:58:32.965 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:32 smithi069 ceph-mon[27071]: [26/Mar/2024:01:58:31] ENGINE Bus STARTING 2024-03-26T01:58:32.965 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:32 smithi069 ceph-mon[27071]: [26/Mar/2024:01:58:31] ENGINE Serving on https://172.21.15.19:7150 2024-03-26T01:58:32.965 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:32 smithi069 ceph-mon[27071]: from='client.34108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:58:32.965 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:32 smithi069 ceph-mon[27071]: pgmap v3: 97 pgs: 97 active+clean; 331 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-03-26T01:58:32.965 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:32 smithi069 ceph-mon[27071]: [26/Mar/2024:01:58:31] ENGINE Serving on http://172.21.15.19:8765 2024-03-26T01:58:32.965 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:32 smithi069 ceph-mon[27071]: [26/Mar/2024:01:58:31] ENGINE Bus STARTED 2024-03-26T01:58:33.103 INFO:teuthology.orchestra.run.smithi019.stdout: 5 158422 12.05 MB/sec execute 186 sec latency 59.035 ms 2024-03-26T01:58:33.473 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T01:58:33.473 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": { 2024-03-26T01:58:33.473 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-26T01:58:33.473 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 1 2024-03-26T01:58:33.473 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:58:33.473 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": { 2024-03-26T01:58:33.474 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T01:58:33.474 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:58:33.474 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": { 2024-03-26T01:58:33.474 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T01:58:33.474 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:58:33.474 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": { 2024-03-26T01:58:33.474 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T01:58:33.474 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:58:33.474 INFO:teuthology.orchestra.run.smithi019.stdout: "overall": { 2024-03-26T01:58:33.474 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 11, 2024-03-26T01:58:33.474 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 3 2024-03-26T01:58:33.474 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-26T01:58:33.474 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T01:58:33.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:33 smithi019 ceph-mon[66359]: from='client.34126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:58:33.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:33 smithi019 ceph-mon[66359]: pgmap v4: 97 pgs: 97 active+clean; 331 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-03-26T01:58:33.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:33 smithi019 ceph-mon[66359]: from='client.34128 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:58:33.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:33 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/1694474301' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T01:58:33.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:33 smithi019 ceph-mon[66359]: mgrmap e39: smithi019.ixuvdm(active, since 3s), standbys: smithi069.wpbewu 2024-03-26T01:58:33.979 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:33 smithi069 ceph-mon[27071]: from='client.34126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:58:33.979 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:33 smithi069 ceph-mon[27071]: pgmap v4: 97 pgs: 97 active+clean; 331 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-03-26T01:58:33.979 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:33 smithi069 ceph-mon[27071]: from='client.34128 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:58:33.979 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:33 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/1694474301' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T01:58:33.979 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:33 smithi069 ceph-mon[27071]: mgrmap e39: smithi019.ixuvdm(active, since 3s), standbys: smithi069.wpbewu 2024-03-26T01:58:34.036 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T01:58:34.036 INFO:teuthology.orchestra.run.smithi019.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2", 2024-03-26T01:58:34.036 INFO:teuthology.orchestra.run.smithi019.stdout: "in_progress": true, 2024-03-26T01:58:34.036 INFO:teuthology.orchestra.run.smithi019.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T01:58:34.036 INFO:teuthology.orchestra.run.smithi019.stdout: "services_complete": [ 2024-03-26T01:58:34.036 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr" 2024-03-26T01:58:34.036 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-26T01:58:34.036 INFO:teuthology.orchestra.run.smithi019.stdout: "progress": "2/23 daemons upgraded", 2024-03-26T01:58:34.037 INFO:teuthology.orchestra.run.smithi019.stdout: "message": "", 2024-03-26T01:58:34.037 INFO:teuthology.orchestra.run.smithi019.stdout: "is_paused": false 2024-03-26T01:58:34.037 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T01:58:34.103 INFO:teuthology.orchestra.run.smithi019.stdout: 5 158814 12.02 MB/sec execute 187 sec latency 44.109 ms 2024-03-26T01:58:34.686 INFO:teuthology.orchestra.run.smithi019.stdout:HEALTH_OK 2024-03-26T01:58:35.103 INFO:teuthology.orchestra.run.smithi019.stdout: 5 159186 12.04 MB/sec execute 188 sec latency 44.706 ms 2024-03-26T01:58:35.791 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:35 smithi019 ceph-mon[66359]: from='client.24647 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:58:35.792 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:35 smithi019 ceph-mon[66359]: pgmap v5: 97 pgs: 97 active+clean; 331 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-03-26T01:58:35.792 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:35 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/3765486833' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T01:58:35.792 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:35 smithi019 ceph-mon[66359]: mgrmap e40: smithi019.ixuvdm(active, since 4s), standbys: smithi069.wpbewu 2024-03-26T01:58:35.792 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:35 smithi019 ceph-mon[66359]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:35.792 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:35 smithi019 ceph-mon[66359]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:36.103 INFO:teuthology.orchestra.run.smithi019.stdout: 5 159766 12.08 MB/sec execute 189 sec latency 32.196 ms 2024-03-26T01:58:36.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:35 smithi069 ceph-mon[27071]: from='client.24647 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:58:36.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:35 smithi069 ceph-mon[27071]: pgmap v5: 97 pgs: 97 active+clean; 331 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-03-26T01:58:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:35 smithi069 ceph-mon[27071]: from='client.? 172.21.15.19:0/3765486833' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T01:58:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:35 smithi069 ceph-mon[27071]: mgrmap e40: smithi019.ixuvdm(active, since 4s), standbys: smithi069.wpbewu 2024-03-26T01:58:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:35 smithi069 ceph-mon[27071]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:35 smithi069 ceph-mon[27071]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:37.103 INFO:teuthology.orchestra.run.smithi019.stdout: 5 160294 12.09 MB/sec execute 190 sec latency 23.345 ms 2024-03-26T01:58:37.550 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:37 smithi019 ceph-mon[66359]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:37.805 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:37 smithi019 ceph-mon[66359]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:37.805 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:37 smithi019 ceph-mon[66359]: pgmap v6: 97 pgs: 97 active+clean; 335 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 1.5 MiB/s wr, 64 op/s 2024-03-26T01:58:37.837 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:37 smithi069 ceph-mon[27071]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:37.837 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:37 smithi069 ceph-mon[27071]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:37.837 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:37 smithi069 ceph-mon[27071]: pgmap v6: 97 pgs: 97 active+clean; 335 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 1.5 MiB/s wr, 64 op/s 2024-03-26T01:58:38.103 INFO:teuthology.orchestra.run.smithi019.stdout: 5 160897 12.08 MB/sec execute 191 sec latency 26.250 ms 2024-03-26T01:58:39.103 INFO:teuthology.orchestra.run.smithi019.stdout: 5 161432 12.07 MB/sec execute 192 sec latency 35.375 ms 2024-03-26T01:58:39.836 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:39 smithi019 ceph-mon[66359]: pgmap v7: 97 pgs: 97 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 35 KiB/s rd, 4.8 MiB/s wr, 450 op/s 2024-03-26T01:58:39.999 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:39 smithi069 ceph-mon[27071]: pgmap v7: 97 pgs: 97 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 35 KiB/s rd, 4.8 MiB/s wr, 450 op/s 2024-03-26T01:58:40.103 INFO:teuthology.orchestra.run.smithi019.stdout: 5 161944 12.10 MB/sec execute 193 sec latency 36.718 ms 2024-03-26T01:58:41.103 INFO:teuthology.orchestra.run.smithi019.stdout: 5 162564 12.13 MB/sec execute 194 sec latency 31.359 ms 2024-03-26T01:58:41.609 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:41 smithi019 ceph-mon[66359]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:41.609 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:41 smithi019 ceph-mon[66359]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:41.610 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:41 smithi019 ceph-mon[66359]: pgmap v8: 97 pgs: 97 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 3.8 MiB/s wr, 350 op/s 2024-03-26T01:58:41.610 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:41 smithi019 ceph-mon[66359]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:41.610 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:41 smithi019 ceph-mon[66359]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:41.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:41 smithi069 ceph-mon[27071]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:41.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:41 smithi069 ceph-mon[27071]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:41.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:41 smithi069 ceph-mon[27071]: pgmap v8: 97 pgs: 97 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 3.8 MiB/s wr, 350 op/s 2024-03-26T01:58:41.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:41 smithi069 ceph-mon[27071]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:41.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:41 smithi069 ceph-mon[27071]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:42.103 INFO:teuthology.orchestra.run.smithi019.stdout: 5 162897 12.13 MB/sec execute 195 sec latency 34.564 ms 2024-03-26T01:58:43.103 INFO:teuthology.orchestra.run.smithi019.stdout: 5 163225 12.13 MB/sec execute 196 sec latency 41.419 ms 2024-03-26T01:58:43.896 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:43 smithi019 ceph-mon[66359]: pgmap v9: 97 pgs: 97 active+clean; 333 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 28 KiB/s rd, 6.6 MiB/s wr, 644 op/s 2024-03-26T01:58:43.897 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:43 smithi019 ceph-mon[66359]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:43.897 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:43 smithi019 ceph-mon[66359]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:43.897 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:43 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:43.897 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:43 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:58:43.897 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:43 smithi019 ceph-mon[66359]: Updating smithi019:/etc/ceph/ceph.conf 2024-03-26T01:58:43.897 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:43 smithi019 ceph-mon[66359]: Updating smithi069:/etc/ceph/ceph.conf 2024-03-26T01:58:44.103 INFO:teuthology.orchestra.run.smithi019.stdout: 5 163526 12.10 MB/sec execute 197 sec latency 74.441 ms 2024-03-26T01:58:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:43 smithi069 ceph-mon[27071]: pgmap v9: 97 pgs: 97 active+clean; 333 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 28 KiB/s rd, 6.6 MiB/s wr, 644 op/s 2024-03-26T01:58:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:43 smithi069 ceph-mon[27071]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:43 smithi069 ceph-mon[27071]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:43 smithi069 ceph-mon[27071]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:43 smithi069 ceph-mon[27071]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:58:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:43 smithi069 ceph-mon[27071]: Updating smithi019:/etc/ceph/ceph.conf 2024-03-26T01:58:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:43 smithi069 ceph-mon[27071]: Updating smithi069:/etc/ceph/ceph.conf 2024-03-26T01:58:45.103 INFO:teuthology.orchestra.run.smithi019.stdout: 5 163765 12.07 MB/sec execute 198 sec latency 52.220 ms 2024-03-26T01:58:45.171 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:44 smithi019 ceph-mon[66359]: Updating smithi069:/var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:58:45.171 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:44 smithi019 ceph-mon[66359]: Updating smithi019:/var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:58:45.171 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:44 smithi019 ceph-mon[66359]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:58:45.172 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:44 smithi019 ceph-mon[66359]: Updating smithi019:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:58:45.172 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:44 smithi019 ceph-mon[66359]: Updating smithi069:/var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T01:58:45.172 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:44 smithi019 ceph-mon[66359]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:45.172 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:44 smithi019 ceph-mon[66359]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:45.172 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:44 smithi019 ceph-mon[66359]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:45.172 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:44 smithi019 ceph-mon[66359]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:45.172 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:44 smithi019 ceph-mon[66359]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:45.172 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:44 smithi019 ceph-mon[66359]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:45.172 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:44 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:45.172 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:44 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T01:58:45.172 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:44 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-26T01:58:45.228 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:44 smithi069 ceph-mon[27071]: Updating smithi069:/var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:58:45.228 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:44 smithi069 ceph-mon[27071]: Updating smithi019:/var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:58:45.228 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:44 smithi069 ceph-mon[27071]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:58:45.228 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:44 smithi069 ceph-mon[27071]: Updating smithi019:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:58:45.228 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:44 smithi069 ceph-mon[27071]: Updating smithi069:/var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T01:58:45.228 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:44 smithi069 ceph-mon[27071]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:45.228 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:44 smithi069 ceph-mon[27071]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:45.229 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:44 smithi069 ceph-mon[27071]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:45.229 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:44 smithi069 ceph-mon[27071]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:45.229 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:44 smithi069 ceph-mon[27071]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:45.229 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:44 smithi069 ceph-mon[27071]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:45.229 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:44 smithi069 ceph-mon[27071]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:45.229 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:44 smithi069 ceph-mon[27071]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T01:58:45.229 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:44 smithi069 ceph-mon[27071]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-26T01:58:46.103 INFO:teuthology.orchestra.run.smithi019.stdout: 5 163978 12.03 MB/sec execute 199 sec latency 52.932 ms 2024-03-26T01:58:46.187 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:45 smithi019 ceph-mon[66359]: Updating smithi019:/var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T01:58:46.187 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:45 smithi019 ceph-mon[66359]: pgmap v10: 97 pgs: 97 active+clean; 333 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 6.0 MiB/s wr, 590 op/s 2024-03-26T01:58:46.187 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:58:46.231 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:45 smithi069 ceph-mon[27071]: Updating smithi019:/var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T01:58:46.231 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:45 smithi069 ceph-mon[27071]: pgmap v10: 97 pgs: 97 active+clean; 333 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 6.0 MiB/s wr, 590 op/s 2024-03-26T01:58:46.231 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:45 smithi069 ceph-mon[27071]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:58:47.103 INFO:teuthology.orchestra.run.smithi019.stdout: 5 164233 12.01 MB/sec execute 200 sec latency 62.085 ms 2024-03-26T01:58:47.466 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:47 smithi019 ceph-mon[66359]: Upgrade: Updating mon.smithi069 2024-03-26T01:58:47.466 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:47 smithi019 ceph-mon[66359]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:47.467 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:47 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:58:47.467 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:47 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T01:58:47.467 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:47 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:47.467 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:47 smithi019 ceph-mon[66359]: Deploying daemon mon.smithi069 on smithi069 2024-03-26T01:58:47.467 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:47 smithi019 ceph-mon[66359]: pgmap v11: 97 pgs: 97 active+clean; 332 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 6.8 MiB/s wr, 623 op/s 2024-03-26T01:58:47.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:47 smithi069 ceph-mon[27071]: Upgrade: Updating mon.smithi069 2024-03-26T01:58:47.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:47 smithi069 ceph-mon[27071]: from='mgr.34104 ' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:47.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:47 smithi069 ceph-mon[27071]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:58:47.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:47 smithi069 ceph-mon[27071]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T01:58:47.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:47 smithi069 ceph-mon[27071]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:47.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:47 smithi069 ceph-mon[27071]: Deploying daemon mon.smithi069 on smithi069 2024-03-26T01:58:47.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:47 smithi069 ceph-mon[27071]: pgmap v11: 97 pgs: 97 active+clean; 332 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 6.8 MiB/s wr, 623 op/s 2024-03-26T01:58:48.103 INFO:teuthology.orchestra.run.smithi019.stdout: 5 164718 11.98 MB/sec execute 201 sec latency 54.702 ms 2024-03-26T01:58:48.736 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:48 smithi069 systemd[1]: Stopping Ceph mon.smithi069 for 692607c6-eb12-11ee-95d0-87774f69a715... 2024-03-26T01:58:49.077 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:48 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi069[27054]: 2024-03-26T01:58:48.734+0000 7f93e337f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi069 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T01:58:49.077 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:48 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi069[27054]: 2024-03-26T01:58:48.734+0000 7f93e337f700 -1 mon.smithi069@1(peon) e2 *** Got Signal Terminated *** 2024-03-26T01:58:49.078 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:48 smithi069 podman[52930]: 2024-03-26 01:58:48.755968779 +0000 UTC m=+0.140578055 container died 2d51b7b7bb325126db6627f085a1b2569521a8027f84e5928a345717b90c321c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi069, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, distribution-scope=public, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, name=centos-stream, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-03-26T01:58:49.103 INFO:teuthology.orchestra.run.smithi019.stdout: 5 165330 12.01 MB/sec execute 202 sec latency 25.307 ms 2024-03-26T01:58:49.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:49 smithi069 podman[52930]: 2024-03-26 01:58:49.169378876 +0000 UTC m=+0.553988150 container cleanup 2d51b7b7bb325126db6627f085a1b2569521a8027f84e5928a345717b90c321c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi069, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., release=754, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-03-26T01:58:49.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:49 smithi069 bash[52930]: ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi069 2024-03-26T01:58:49.749 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:49 smithi069 podman[52943]: 2024-03-26 01:58:49.420819093 +0000 UTC m=+0.661007961 container remove 2d51b7b7bb325126db6627f085a1b2569521a8027f84e5928a345717b90c321c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi069, vcs-type=git, RELEASE=HEAD, GIT_BRANCH=HEAD, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, release=754, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, vendor=Red Hat, Inc., ceph=True, com.redhat.component=centos-stream-container, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-03-26T01:58:49.749 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:49 smithi069 systemd[1]: ceph-692607c6-eb12-11ee-95d0-87774f69a715@mon.smithi069.service: Deactivated successfully. 2024-03-26T01:58:49.749 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:49 smithi069 systemd[1]: Stopped Ceph mon.smithi069 for 692607c6-eb12-11ee-95d0-87774f69a715. 2024-03-26T01:58:49.749 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:49 smithi069 systemd[1]: ceph-692607c6-eb12-11ee-95d0-87774f69a715@mon.smithi069.service: Consumed 7.262s CPU time. 2024-03-26T01:58:50.103 INFO:teuthology.orchestra.run.smithi019.stdout: 5 165959 12.03 MB/sec execute 203 sec latency 40.260 ms 2024-03-26T01:58:50.141 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:49 smithi069 systemd[1]: Starting Ceph mon.smithi069 for 692607c6-eb12-11ee-95d0-87774f69a715... 2024-03-26T01:58:50.458 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 podman[53068]: 2024-03-26 01:58:50.046498912 +0000 UTC m=+0.024760107 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T01:58:50.458 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 podman[53068]: 2024-03-26 01:58:50.173189863 +0000 UTC m=+0.151451039 container create a856bd263104279c177520817cefc196208cf47d085ab195d3cd798b608d2301 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi069, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-26T01:58:50.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 podman[53068]: 2024-03-26 01:58:50.458961721 +0000 UTC m=+0.437222898 container init a856bd263104279c177520817cefc196208cf47d085ab195d3cd798b608d2301 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi069, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T01:58:50.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 podman[53068]: 2024-03-26 01:58:50.464371528 +0000 UTC m=+0.442632711 container start a856bd263104279c177520817cefc196208cf47d085ab195d3cd798b608d2301 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi069, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T01:58:50.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T01:58:50.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev), process ceph-mon, pid 2 2024-03-26T01:58:50.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: pidfile_write: ignore empty --pid-file 2024-03-26T01:58:50.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: load: jerasure load: lrc 2024-03-26T01:58:50.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: RocksDB version: 7.9.2 2024-03-26T01:58:50.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Git sha 0 2024-03-26T01:58:50.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Compile date 2024-03-25 16:14:20 2024-03-26T01:58:50.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: DB SUMMARY 2024-03-26T01:58:50.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: DB Session ID: 4OSPEFB60RE0QHFBM4HH 2024-03-26T01:58:50.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: CURRENT file: CURRENT 2024-03-26T01:58:50.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T01:58:50.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: MANIFEST file: MANIFEST-000009 size: 903 Bytes 2024-03-26T01:58:50.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi069/store.db dir, Total Num: 1, files: 000024.sst 2024-03-26T01:58:50.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi069/store.db: 000022.log size: 7584231 ; 2024-03-26T01:58:50.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.error_if_exists: 0 2024-03-26T01:58:50.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.create_if_missing: 0 2024-03-26T01:58:50.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.paranoid_checks: 1 2024-03-26T01:58:50.861 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T01:58:50.861 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T01:58:50.861 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T01:58:50.861 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.env: 0x561842ae9ae0 2024-03-26T01:58:50.861 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T01:58:50.861 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.info_log: 0x5618438cb7c0 2024-03-26T01:58:50.861 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T01:58:50.862 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.statistics: (nil) 2024-03-26T01:58:50.863 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.use_fsync: 0 2024-03-26T01:58:50.863 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.max_log_file_size: 0 2024-03-26T01:58:50.863 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T01:58:50.863 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T01:58:50.863 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T01:58:50.863 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T01:58:50.863 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.allow_fallocate: 1 2024-03-26T01:58:50.863 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T01:58:50.863 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T01:58:50.863 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.use_direct_reads: 0 2024-03-26T01:58:50.863 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T01:58:50.863 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T01:58:50.864 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.db_log_dir: 2024-03-26T01:58:50.864 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.wal_dir: 2024-03-26T01:58:50.864 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T01:58:50.864 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T01:58:50.864 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T01:58:50.864 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T01:58:50.864 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T01:58:50.864 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T01:58:50.864 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T01:58:50.864 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T01:58:50.864 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.write_buffer_manager: 0x56184389e1e0 2024-03-26T01:58:50.864 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T01:58:50.864 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T01:58:50.864 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T01:58:50.865 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T01:58:50.865 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T01:58:50.865 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T01:58:50.865 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T01:58:50.865 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T01:58:50.865 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.unordered_write: 0 2024-03-26T01:58:50.865 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T01:58:50.865 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T01:58:50.865 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T01:58:50.865 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T01:58:50.865 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.row_cache: None 2024-03-26T01:58:50.865 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.wal_filter: None 2024-03-26T01:58:50.865 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T01:58:50.865 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T01:58:50.866 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.two_write_queues: 0 2024-03-26T01:58:50.866 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T01:58:50.866 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.wal_compression: 0 2024-03-26T01:58:50.866 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.atomic_flush: 0 2024-03-26T01:58:50.866 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T01:58:50.866 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T01:58:50.866 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T01:58:50.866 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.log_readahead_size: 0 2024-03-26T01:58:50.866 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T01:58:50.866 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T01:58:50.866 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T01:58:50.866 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T01:58:50.866 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T01:58:50.867 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T01:58:50.867 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T01:58:50.867 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.max_background_jobs: 2 2024-03-26T01:58:50.867 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.max_background_compactions: -1 2024-03-26T01:58:50.867 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.max_subcompactions: 1 2024-03-26T01:58:50.867 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T01:58:50.867 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T01:58:50.867 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T01:58:50.867 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T01:58:50.867 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T01:58:50.867 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T01:58:50.867 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T01:58:50.867 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T01:58:50.867 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.max_open_files: -1 2024-03-26T01:58:50.868 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T01:58:50.868 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T01:58:50.868 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T01:58:50.868 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T01:58:50.868 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.max_background_flushes: -1 2024-03-26T01:58:50.868 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Compression algorithms supported: 2024-03-26T01:58:50.868 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T01:58:50.868 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: kZSTD supported: 0 2024-03-26T01:58:50.868 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: kXpressCompression supported: 0 2024-03-26T01:58:50.868 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T01:58:50.868 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: kZlibCompression supported: 1 2024-03-26T01:58:50.868 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: kSnappyCompression supported: 1 2024-03-26T01:58:50.868 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: kLZ4Compression supported: 1 2024-03-26T01:58:50.869 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: kBZip2Compression supported: 0 2024-03-26T01:58:50.869 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T01:58:50.869 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T01:58:50.869 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi069/store.db/MANIFEST-000009 2024-03-26T01:58:50.869 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T01:58:50.869 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T01:58:50.869 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.merge_operator: 2024-03-26T01:58:50.869 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.compaction_filter: None 2024-03-26T01:58:50.869 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T01:58:50.869 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T01:58:50.870 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T01:58:50.870 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T01:58:50.870 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5618438cb8e0) 2024-03-26T01:58:50.870 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: cache_index_and_filter_blocks: 1 2024-03-26T01:58:50.870 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T01:58:50.870 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T01:58:50.870 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: pin_top_level_index_and_filter: 1 2024-03-26T01:58:50.870 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: index_type: 0 2024-03-26T01:58:50.870 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: data_block_index_type: 0 2024-03-26T01:58:50.870 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: index_shortening: 1 2024-03-26T01:58:50.870 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T01:58:50.870 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: checksum: 4 2024-03-26T01:58:50.870 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: no_block_cache: 0 2024-03-26T01:58:50.870 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: block_cache: 0x5618438cf090 2024-03-26T01:58:50.871 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: block_cache_name: BinnedLRUCache 2024-03-26T01:58:50.871 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: block_cache_options: 2024-03-26T01:58:50.871 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: capacity : 536870912 2024-03-26T01:58:50.871 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: num_shard_bits : 4 2024-03-26T01:58:50.871 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: strict_capacity_limit : 0 2024-03-26T01:58:50.871 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: high_pri_pool_ratio: 0.000 2024-03-26T01:58:50.871 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: block_cache_compressed: (nil) 2024-03-26T01:58:50.871 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: persistent_cache: (nil) 2024-03-26T01:58:50.871 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: block_size: 4096 2024-03-26T01:58:50.871 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: block_size_deviation: 10 2024-03-26T01:58:50.871 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: block_restart_interval: 16 2024-03-26T01:58:50.871 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: index_block_restart_interval: 1 2024-03-26T01:58:50.871 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: metadata_block_size: 4096 2024-03-26T01:58:50.871 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: partition_filters: 0 2024-03-26T01:58:50.872 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: use_delta_encoding: 1 2024-03-26T01:58:50.872 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: filter_policy: bloomfilter 2024-03-26T01:58:50.872 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: whole_key_filtering: 1 2024-03-26T01:58:50.872 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: verify_compression: 0 2024-03-26T01:58:50.872 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: read_amp_bytes_per_bit: 0 2024-03-26T01:58:50.872 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: format_version: 5 2024-03-26T01:58:50.872 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: enable_index_compression: 1 2024-03-26T01:58:50.872 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: block_align: 0 2024-03-26T01:58:50.872 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: max_auto_readahead_size: 262144 2024-03-26T01:58:50.872 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: prepopulate_block_cache: 0 2024-03-26T01:58:50.872 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: initial_auto_readahead_size: 8192 2024-03-26T01:58:50.872 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T01:58:50.872 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T01:58:50.872 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T01:58:50.872 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.compression: NoCompression 2024-03-26T01:58:50.873 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T01:58:50.873 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T01:58:50.873 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T01:58:50.873 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.num_levels: 7 2024-03-26T01:58:50.873 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T01:58:50.873 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T01:58:50.873 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T01:58:50.873 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T01:58:50.873 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T01:58:50.873 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T01:58:50.873 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T01:58:50.873 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T01:58:50.873 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T01:58:50.874 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T01:58:50.874 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T01:58:50.874 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T01:58:50.874 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T01:58:50.874 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T01:58:50.874 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T01:58:50.874 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T01:58:50.874 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T01:58:50.874 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T01:58:50.874 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T01:58:50.874 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T01:58:50.874 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T01:58:50.874 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T01:58:50.874 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T01:58:50.875 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T01:58:50.875 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T01:58:50.875 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T01:58:50.875 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T01:58:50.875 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T01:58:50.875 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T01:58:50.875 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T01:58:50.875 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T01:58:50.875 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T01:58:50.875 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T01:58:50.875 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T01:58:50.875 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T01:58:50.875 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T01:58:50.875 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T01:58:50.875 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T01:58:50.876 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T01:58:50.876 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T01:58:50.876 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T01:58:50.876 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T01:58:50.876 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T01:58:50.876 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T01:58:50.876 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T01:58:50.876 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T01:58:50.876 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T01:58:50.876 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T01:58:50.876 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T01:58:50.876 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T01:58:50.876 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T01:58:50.877 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T01:58:50.877 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T01:58:50.877 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.table_properties_collectors: 2024-03-26T01:58:50.877 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.inplace_update_support: 0 2024-03-26T01:58:50.877 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T01:58:50.877 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T01:58:50.877 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T01:58:50.877 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T01:58:50.877 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.bloom_locality: 0 2024-03-26T01:58:50.877 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.max_successive_merges: 0 2024-03-26T01:58:50.878 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T01:58:50.878 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T01:58:50.878 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T01:58:50.878 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T01:58:50.878 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.ttl: 2592000 2024-03-26T01:58:50.878 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T01:58:50.878 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T01:58:50.878 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T01:58:50.878 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.enable_blob_files: false 2024-03-26T01:58:50.878 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.min_blob_size: 0 2024-03-26T01:58:50.878 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T01:58:50.878 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T01:58:50.879 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T01:58:50.879 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T01:58:50.879 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T01:58:50.879 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T01:58:50.879 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T01:58:50.879 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T01:58:50.879 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 24.sst 2024-03-26T01:58:50.879 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-26T01:58:50.879 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi069/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 26, last_sequence is 12209, log_number is 22,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-26T01:58:50.879 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 22 2024-03-26T01:58:50.879 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: c96e3886-97e8-4007-8920-fcee6090575d 2024-03-26T01:58:50.879 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418330508129, "job": 1, "event": "recovery_started", "wal_files": [22]} 2024-03-26T01:58:50.879 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #22 mode 2 2024-03-26T01:58:50.879 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418330529331, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 27, "file_size": 4613709, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12214, "largest_seqno": 14087, "table_properties": {"data_size": 4605585, "index_size": 5248, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1861, "raw_key_size": 16841, "raw_average_key_size": 22, "raw_value_size": 4589675, "raw_average_value_size": 6252, "num_data_blocks": 238, "num_entries": 734, "num_filter_entries": 734, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711418330, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "c96e3886-97e8-4007-8920-fcee6090575d", "db_session_id": "4OSPEFB60RE0QHFBM4HH", "orig_file_number": 27, "seqno_to_time_mapping": "N/A"}} 2024-03-26T01:58:50.880 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418330529429, "job": 1, "event": "recovery_finished"} 2024-03-26T01:58:50.880 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: [db/version_set.cc:5047] Creating manifest 29 2024-03-26T01:58:50.880 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-26T01:58:50.880 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi069/store.db/000022.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T01:58:50.880 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5618439be000 2024-03-26T01:58:50.880 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: DB pointer 0x5618439aa000 2024-03-26T01:58:50.880 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T01:58:50.880 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T01:58:50.880 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: ** DB Stats ** 2024-03-26T01:58:50.880 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T01:58:50.880 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T01:58:50.880 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T01:58:50.880 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T01:58:50.880 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T01:58:50.880 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T01:58:50.881 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T01:58:50.881 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: 2024-03-26T01:58:50.881 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: ** Compaction Stats [default] ** 2024-03-26T01:58:50.881 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T01:58:50.881 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T01:58:50.881 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: L0 1/0 4.40 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 477.0 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-03-26T01:58:50.881 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: L6 1/0 6.25 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-26T01:58:50.881 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: Sum 2/0 10.65 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 477.0 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-03-26T01:58:50.881 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 477.0 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-03-26T01:58:50.881 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: 2024-03-26T01:58:50.881 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: ** Compaction Stats [default] ** 2024-03-26T01:58:50.881 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T01:58:50.881 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T01:58:50.881 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 477.0 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-03-26T01:58:50.881 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: 2024-03-26T01:58:50.882 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T01:58:50.882 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: 2024-03-26T01:58:50.882 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T01:58:50.882 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: Flush(GB): cumulative 0.004, interval 0.004 2024-03-26T01:58:50.882 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T01:58:50.882 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T01:58:50.882 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T01:58:50.882 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T01:58:50.882 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: Cumulative compaction: 0.00 GB write, 161.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T01:58:50.882 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: Interval compaction: 0.00 GB write, 161.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T01:58:50.882 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T01:58:50.883 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: Block cache BinnedLRUCache@0x5618438cf090#2 capacity: 512.00 MB usage: 37.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.4e-05 secs_since: 0 2024-03-26T01:58:50.883 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: Block cache entry stats(count,size,portion): DataBlock(1,4.08 KB,0.000777841%) FilterBlock(2,11.06 KB,0.00211%) IndexBlock(2,22.16 KB,0.00422597%) Misc(1,0.00 KB,0%) 2024-03-26T01:58:50.883 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: 2024-03-26T01:58:50.883 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T01:58:50.883 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: starting mon.smithi069 rank 1 at public addrs [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] at bind addrs [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi069 fsid 692607c6-eb12-11ee-95d0-87774f69a715 2024-03-26T01:58:50.883 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: mon.smithi069@-1(???) e2 preinit fsid 692607c6-eb12-11ee-95d0-87774f69a715 2024-03-26T01:58:50.883 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: mon.smithi069@-1(???).mds e7 new map 2024-03-26T01:58:50.883 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: mon.smithi069@-1(???).mds e7 print_map 2024-03-26T01:58:50.883 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: e7 2024-03-26T01:58:50.883 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T01:58:50.883 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T01:58:50.883 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: legacy client fscid: 1 2024-03-26T01:58:50.883 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: 2024-03-26T01:58:50.884 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: Filesystem 'foofs' (1) 2024-03-26T01:58:50.884 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: fs_name foofs 2024-03-26T01:58:50.884 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: epoch 5 2024-03-26T01:58:50.884 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-26T01:58:50.884 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: created 2024-03-26T01:52:48.813757+0000 2024-03-26T01:58:50.884 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: modified 2024-03-26T01:52:54.832475+0000 2024-03-26T01:58:50.884 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: tableserver 0 2024-03-26T01:58:50.884 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: root 0 2024-03-26T01:58:50.884 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: session_timeout 60 2024-03-26T01:58:50.884 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: session_autoclose 300 2024-03-26T01:58:50.884 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: max_file_size 1099511627776 2024-03-26T01:58:50.884 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: max_xattr_size 65536 2024-03-26T01:58:50.884 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: required_client_features {} 2024-03-26T01:58:50.885 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: last_failure 0 2024-03-26T01:58:50.885 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: last_failure_osd_epoch 0 2024-03-26T01:58:50.885 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T01:58:50.885 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: max_mds 1 2024-03-26T01:58:50.885 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: in 0 2024-03-26T01:58:50.885 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: up {0=14478} 2024-03-26T01:58:50.885 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: failed 2024-03-26T01:58:50.885 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: damaged 2024-03-26T01:58:50.885 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: stopped 2024-03-26T01:58:50.885 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: data_pools [3] 2024-03-26T01:58:50.885 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: metadata_pool 2 2024-03-26T01:58:50.885 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: inline_data disabled 2024-03-26T01:58:50.885 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: balancer 2024-03-26T01:58:50.886 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: bal_rank_mask -1 2024-03-26T01:58:50.886 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: standby_count_wanted 1 2024-03-26T01:58:50.886 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: qdb_cluster leader: 0 members: 2024-03-26T01:58:50.886 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: [mds.foofs.smithi019.lmryom{0:14478} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.19:6834/3267072550,v1:172.21.15.19:6835/3267072550] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T01:58:50.886 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: 2024-03-26T01:58:50.886 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: 2024-03-26T01:58:50.886 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: Standby daemons: 2024-03-26T01:58:50.886 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: 2024-03-26T01:58:50.887 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout: [mds.foofs.smithi069.spgwgo{-1:24309} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.69:6832/641242727,v1:172.21.15.69:6833/641242727] compat {c=[1],r=[1],i=[7ff]}] 2024-03-26T01:58:50.887 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: mon.smithi069@-1(???).osd e73 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-26T01:58:50.887 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: mon.smithi069@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-03-26T01:58:50.887 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: mon.smithi069@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-03-26T01:58:50.887 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: mon.smithi069@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-03-26T01:58:50.887 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 ceph-mon[53087]: mon.smithi069@-1(???).paxosservice(auth 1..30) refresh upgraded, format 0 -> 3 2024-03-26T01:58:50.887 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 bash[53068]: a856bd263104279c177520817cefc196208cf47d085ab195d3cd798b608d2301 2024-03-26T01:58:50.887 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:50 smithi069 systemd[1]: Started Ceph mon.smithi069 for 692607c6-eb12-11ee-95d0-87774f69a715. 2024-03-26T01:58:51.103 INFO:teuthology.orchestra.run.smithi019.stdout: 5 166749 12.12 MB/sec execute 204 sec latency 26.660 ms 2024-03-26T01:58:51.992 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:51 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi019"}]: dispatch 2024-03-26T01:58:51.992 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:51 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi069"}]: dispatch 2024-03-26T01:58:52.103 INFO:teuthology.orchestra.run.smithi019.stdout: 5 167380 12.14 MB/sec execute 205 sec latency 29.088 ms 2024-03-26T01:58:52.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:51 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi019"}]: dispatch 2024-03-26T01:58:52.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:51 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi069"}]: dispatch 2024-03-26T01:58:52.998 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:52 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi019"}]: dispatch 2024-03-26T01:58:52.999 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:52 smithi019 ceph-mon[66359]: mon.smithi019 calling monitor election 2024-03-26T01:58:52.999 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:52 smithi019 ceph-mon[66359]: mon.smithi069 calling monitor election 2024-03-26T01:58:52.999 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:52 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi069"}]: dispatch 2024-03-26T01:58:52.999 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:52 smithi019 ceph-mon[66359]: mon.smithi019 is new leader, mons smithi019,smithi069 in quorum (ranks 0,1) 2024-03-26T01:58:52.999 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:52 smithi019 ceph-mon[66359]: monmap e3: 2 mons at {smithi019=[v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0],smithi069=[v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T01:58:52.999 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:52 smithi019 ceph-mon[66359]: fsmap foofs:1 {0=foofs.smithi019.lmryom=up:active} 1 up:standby 2024-03-26T01:58:52.999 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:52 smithi019 ceph-mon[66359]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T01:58:52.999 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:52 smithi019 ceph-mon[66359]: mgrmap e40: smithi019.ixuvdm(active, since 21s), standbys: smithi069.wpbewu 2024-03-26T01:58:52.999 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:52 smithi019 ceph-mon[66359]: overall HEALTH_OK 2024-03-26T01:58:52.999 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:52 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:52.999 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:52 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:52.999 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:52 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:53.103 INFO:teuthology.orchestra.run.smithi019.stdout: 5 167936 12.13 MB/sec execute 206 sec latency 21.837 ms 2024-03-26T01:58:53.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:52 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi019"}]: dispatch 2024-03-26T01:58:53.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:52 smithi069 ceph-mon[53087]: mon.smithi019 calling monitor election 2024-03-26T01:58:53.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:52 smithi069 ceph-mon[53087]: mon.smithi069 calling monitor election 2024-03-26T01:58:53.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:52 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mon metadata", "id": "smithi069"}]: dispatch 2024-03-26T01:58:53.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:52 smithi069 ceph-mon[53087]: mon.smithi019 is new leader, mons smithi019,smithi069 in quorum (ranks 0,1) 2024-03-26T01:58:53.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:52 smithi069 ceph-mon[53087]: monmap e3: 2 mons at {smithi019=[v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0],smithi069=[v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T01:58:53.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:52 smithi069 ceph-mon[53087]: fsmap foofs:1 {0=foofs.smithi019.lmryom=up:active} 1 up:standby 2024-03-26T01:58:53.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:52 smithi069 ceph-mon[53087]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T01:58:53.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:52 smithi069 ceph-mon[53087]: mgrmap e40: smithi019.ixuvdm(active, since 21s), standbys: smithi069.wpbewu 2024-03-26T01:58:53.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:52 smithi069 ceph-mon[53087]: overall HEALTH_OK 2024-03-26T01:58:53.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:52 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:53.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:52 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:53.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:52 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:54.104 INFO:teuthology.orchestra.run.smithi019.stdout: 5 168308 12.10 MB/sec execute 207 sec latency 61.388 ms 2024-03-26T01:58:54.105 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:53 smithi019 ceph-mon[66359]: pgmap v14: 97 pgs: 97 active+clean; 343 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.7 MiB/s wr, 854 op/s 2024-03-26T01:58:54.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:53 smithi069 ceph-mon[53087]: pgmap v14: 97 pgs: 97 active+clean; 343 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.7 MiB/s wr, 854 op/s 2024-03-26T01:58:55.104 INFO:teuthology.orchestra.run.smithi019.stdout: 5 168624 12.07 MB/sec execute 208 sec latency 75.233 ms 2024-03-26T01:58:56.104 INFO:teuthology.orchestra.run.smithi019.stdout: 5 168944 12.07 MB/sec execute 209 sec latency 47.190 ms 2024-03-26T01:58:56.138 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:55 smithi019 ceph-mon[66359]: pgmap v15: 97 pgs: 97 active+clean; 343 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.5 MiB/s wr, 526 op/s 2024-03-26T01:58:56.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:55 smithi069 ceph-mon[53087]: pgmap v15: 97 pgs: 97 active+clean; 343 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.5 MiB/s wr, 526 op/s 2024-03-26T01:58:57.104 INFO:teuthology.orchestra.run.smithi019.stdout: 5 169334 12.07 MB/sec execute 210 sec latency 65.502 ms 2024-03-26T01:58:57.165 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:57 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:57.165 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:57 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:57.165 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:57 smithi019 ceph-mon[66359]: pgmap v16: 97 pgs: 97 active+clean; 322 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 6.7 MiB/s wr, 601 op/s 2024-03-26T01:58:57.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:57 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:57.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:57 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:57.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:57 smithi069 ceph-mon[53087]: pgmap v16: 97 pgs: 97 active+clean; 322 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 6.7 MiB/s wr, 601 op/s 2024-03-26T01:58:58.104 INFO:teuthology.orchestra.run.smithi019.stdout: 5 169763 12.08 MB/sec execute 211 sec latency 36.594 ms 2024-03-26T01:58:58.456 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:58 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:58.456 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:58 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:58.608 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:58 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:58.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:58 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:58:59.104 INFO:teuthology.orchestra.run.smithi019.stdout: 5 170051 12.08 MB/sec execute 212 sec latency 40.846 ms 2024-03-26T01:58:59.479 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:58:59 smithi019 ceph-mon[66359]: pgmap v17: 97 pgs: 97 active+clean; 329 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 6.9 MiB/s wr, 751 op/s 2024-03-26T01:58:59.861 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:58:59 smithi069 ceph-mon[53087]: pgmap v17: 97 pgs: 97 active+clean; 329 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 6.9 MiB/s wr, 751 op/s 2024-03-26T01:59:00.104 INFO:teuthology.orchestra.run.smithi019.stdout: 5 170347 12.06 MB/sec execute 213 sec latency 39.561 ms 2024-03-26T01:59:01.104 INFO:teuthology.orchestra.run.smithi019.stdout: 5 170665 12.04 MB/sec execute 214 sec latency 48.051 ms 2024-03-26T01:59:01.763 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:01 smithi019 ceph-mon[66359]: pgmap v18: 97 pgs: 97 active+clean; 329 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.8 MiB/s wr, 605 op/s 2024-03-26T01:59:01.763 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:59:01.763 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:01.764 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:01.764 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:01.764 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:59:01.764 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:01.764 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:01.764 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:59:01.764 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T01:59:01.764 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:02.104 INFO:teuthology.orchestra.run.smithi019.stdout: 5 170910 12.01 MB/sec execute 215 sec latency 63.556 ms 2024-03-26T01:59:02.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:01 smithi069 ceph-mon[53087]: pgmap v18: 97 pgs: 97 active+clean; 329 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.8 MiB/s wr, 605 op/s 2024-03-26T01:59:02.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:59:02.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:02.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:02.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:02.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:59:02.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:02.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:02.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:59:02.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T01:59:02.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:02.800 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:02 smithi019 ceph-mon[66359]: Reconfiguring mon.smithi019 (monmap changed)... 2024-03-26T01:59:02.800 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:02 smithi019 ceph-mon[66359]: Reconfiguring daemon mon.smithi019 on smithi019 2024-03-26T01:59:03.104 INFO:teuthology.orchestra.run.smithi019.stdout: 5 171156 11.99 MB/sec execute 216 sec latency 44.906 ms 2024-03-26T01:59:03.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:02 smithi069 ceph-mon[53087]: Reconfiguring mon.smithi019 (monmap changed)... 2024-03-26T01:59:03.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:02 smithi069 ceph-mon[53087]: Reconfiguring daemon mon.smithi019 on smithi019 2024-03-26T01:59:03.825 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:03 smithi019 ceph-mon[66359]: pgmap v19: 97 pgs: 97 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 8.2 MiB/s wr, 798 op/s 2024-03-26T01:59:04.104 INFO:teuthology.orchestra.run.smithi019.stdout: 5 171472 11.96 MB/sec execute 217 sec latency 47.783 ms 2024-03-26T01:59:04.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:03 smithi069 ceph-mon[53087]: pgmap v19: 97 pgs: 97 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 8.2 MiB/s wr, 798 op/s 2024-03-26T01:59:05.104 INFO:teuthology.orchestra.run.smithi019.stdout: 5 171832 11.93 MB/sec execute 218 sec latency 48.734 ms 2024-03-26T01:59:05.117 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:04 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:05.117 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:04 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:05.117 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:04 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi019.ixuvdm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T01:59:05.117 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:04 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T01:59:05.118 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:04 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:05.268 INFO:teuthology.orchestra.run.smithi019.stdout:true 2024-03-26T01:59:05.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:04 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:05.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:04 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:05.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:04 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi019.ixuvdm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T01:59:05.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:04 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T01:59:05.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:04 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:06.104 INFO:teuthology.orchestra.run.smithi019.stdout: 5 172137 11.89 MB/sec execute 219 sec latency 48.027 ms 2024-03-26T01:59:06.145 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:05 smithi019 ceph-mon[66359]: Reconfiguring mgr.smithi019.ixuvdm (monmap changed)... 2024-03-26T01:59:06.145 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:05 smithi019 ceph-mon[66359]: Reconfiguring daemon mgr.smithi019.ixuvdm on smithi019 2024-03-26T01:59:06.145 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:05 smithi019 ceph-mon[66359]: pgmap v20: 97 pgs: 97 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.5 MiB/s wr, 449 op/s 2024-03-26T01:59:06.306 INFO:teuthology.orchestra.run.smithi019.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T01:59:06.306 INFO:teuthology.orchestra.run.smithi019.stdout:alertmanager.smithi019 smithi019 *:9093,9094 running (3m) 25s ago 12m 16.2M - 0.23.0 ba2b418f427c a6630841c2b2 2024-03-26T01:59:06.306 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi019 smithi019 running (12m) 25s ago 12m 6405k - 17.2.0 e1d6a67b021e 713db693d5c1 2024-03-26T01:59:06.307 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi069 smithi069 running (11m) 10s ago 11m 6405k - 17.2.0 e1d6a67b021e a87d27e2b74b 2024-03-26T01:59:06.307 INFO:teuthology.orchestra.run.smithi019.stdout:grafana.smithi019 smithi019 *:3000 running (2m) 25s ago 11m 45.4M - 8.3.5 dad864ee21e9 a97dea3b5fec 2024-03-26T01:59:06.307 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi019.lmryom smithi019 running (6m) 25s ago 6m 37.8M - 17.2.0 e1d6a67b021e 91a7c2942715 2024-03-26T01:59:06.307 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi069.spgwgo smithi069 running (6m) 10s ago 6m 14.2M - 17.2.0 e1d6a67b021e 94d96338dbe1 2024-03-26T01:59:06.307 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi019.ixuvdm smithi019 *:8443,9283,8765 running (118s) 25s ago 13m 478M - 19.0.0-1775-g5e70124b 5f21c9264136 4460b22c7a56 2024-03-26T01:59:06.307 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi069.wpbewu smithi069 *:8443,9283,8765 running (66s) 10s ago 11m 450M - 19.0.0-1775-g5e70124b 5f21c9264136 380fd33842a8 2024-03-26T01:59:06.307 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi019 smithi019 running (47s) 25s ago 13m 33.6M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 c18934705774 2024-03-26T01:59:06.307 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi069 smithi069 running (16s) 10s ago 11m 24.3M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 a856bd263104 2024-03-26T01:59:06.307 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.0.0.smithi019.qbizxo smithi019 *:2049 running (5m) 25s ago 5m 286M - 4.0 e1d6a67b021e cd6628a9075d 2024-03-26T01:59:06.307 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.1.0.smithi069.uawdhc smithi069 *:2049 running (5m) 10s ago 5m 85.3M - 4.0 e1d6a67b021e 6ee3695e7a10 2024-03-26T01:59:06.307 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi019 smithi019 *:9100 running (2m) 25s ago 11m 13.1M - 1.3.1 1dbe0e931976 0d1d184bec63 2024-03-26T01:59:06.307 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi069 smithi069 *:9100 running (2m) 10s ago 11m 20.9M - 1.3.1 1dbe0e931976 c49a006f94e7 2024-03-26T01:59:06.307 INFO:teuthology.orchestra.run.smithi019.stdout:osd.0 smithi019 running (10m) 25s ago 10m 557M 1937M 17.2.0 e1d6a67b021e c1a797230ab6 2024-03-26T01:59:06.307 INFO:teuthology.orchestra.run.smithi019.stdout:osd.1 smithi019 running (10m) 25s ago 10m 476M 1937M 17.2.0 e1d6a67b021e 31d3489c7188 2024-03-26T01:59:06.308 INFO:teuthology.orchestra.run.smithi019.stdout:osd.2 smithi019 running (9m) 25s ago 9m 355M 1937M 17.2.0 e1d6a67b021e f549115e7685 2024-03-26T01:59:06.308 INFO:teuthology.orchestra.run.smithi019.stdout:osd.3 smithi019 running (9m) 25s ago 9m 489M 1937M 17.2.0 e1d6a67b021e 154af68d03e1 2024-03-26T01:59:06.308 INFO:teuthology.orchestra.run.smithi019.stdout:osd.4 smithi069 running (8m) 10s ago 8m 529M 2704M 17.2.0 e1d6a67b021e 4fd5e853f8bf 2024-03-26T01:59:06.308 INFO:teuthology.orchestra.run.smithi019.stdout:osd.5 smithi069 running (8m) 10s ago 8m 360M 2704M 17.2.0 e1d6a67b021e 932e401d87c9 2024-03-26T01:59:06.308 INFO:teuthology.orchestra.run.smithi019.stdout:osd.6 smithi069 running (8m) 10s ago 8m 546M 2704M 17.2.0 e1d6a67b021e b123f75154c8 2024-03-26T01:59:06.308 INFO:teuthology.orchestra.run.smithi019.stdout:osd.7 smithi069 running (7m) 10s ago 7m 456M 2704M 17.2.0 e1d6a67b021e d175fbc9ff8a 2024-03-26T01:59:06.308 INFO:teuthology.orchestra.run.smithi019.stdout:prometheus.smithi019 smithi019 *:9095 running (73s) 25s ago 10m 52.6M - 2.33.4 514e6a882f6e 29f89db72a07 2024-03-26T01:59:06.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:05 smithi069 ceph-mon[53087]: Reconfiguring mgr.smithi019.ixuvdm (monmap changed)... 2024-03-26T01:59:06.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:05 smithi069 ceph-mon[53087]: Reconfiguring daemon mgr.smithi019.ixuvdm on smithi019 2024-03-26T01:59:06.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:05 smithi069 ceph-mon[53087]: pgmap v20: 97 pgs: 97 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.5 MiB/s wr, 449 op/s 2024-03-26T01:59:07.020 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T01:59:07.020 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": { 2024-03-26T01:59:07.020 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T01:59:07.020 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:59:07.020 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": { 2024-03-26T01:59:07.020 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T01:59:07.020 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:59:07.020 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": { 2024-03-26T01:59:07.021 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T01:59:07.021 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:59:07.021 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": { 2024-03-26T01:59:07.021 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T01:59:07.021 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:59:07.021 INFO:teuthology.orchestra.run.smithi019.stdout: "overall": { 2024-03-26T01:59:07.021 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-03-26T01:59:07.021 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 4 2024-03-26T01:59:07.021 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-26T01:59:07.021 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T01:59:07.104 INFO:teuthology.orchestra.run.smithi019.stdout: 5 172413 11.89 MB/sec execute 220 sec latency 61.169 ms 2024-03-26T01:59:07.197 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:06 smithi019 ceph-mon[66359]: from='client.34148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:07.198 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:06 smithi019 ceph-mon[66359]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:07.198 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:06 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:07.198 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:06 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:07.198 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:06 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi019", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-26T01:59:07.198 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:06 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:07.260 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:06 smithi069 ceph-mon[53087]: from='client.34148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:07.260 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:06 smithi069 ceph-mon[53087]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:07.260 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:06 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:07.260 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:06 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:07.260 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:06 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi019", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-26T01:59:07.260 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:06 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:07.543 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T01:59:07.543 INFO:teuthology.orchestra.run.smithi019.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2", 2024-03-26T01:59:07.543 INFO:teuthology.orchestra.run.smithi019.stdout: "in_progress": true, 2024-03-26T01:59:07.543 INFO:teuthology.orchestra.run.smithi019.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T01:59:07.543 INFO:teuthology.orchestra.run.smithi019.stdout: "services_complete": [ 2024-03-26T01:59:07.543 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr", 2024-03-26T01:59:07.543 INFO:teuthology.orchestra.run.smithi019.stdout: "mon" 2024-03-26T01:59:07.543 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-26T01:59:07.543 INFO:teuthology.orchestra.run.smithi019.stdout: "progress": "4/23 daemons upgraded", 2024-03-26T01:59:07.544 INFO:teuthology.orchestra.run.smithi019.stdout: "message": "Currently upgrading mon daemons", 2024-03-26T01:59:07.544 INFO:teuthology.orchestra.run.smithi019.stdout: "is_paused": false 2024-03-26T01:59:07.544 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T01:59:08.104 INFO:teuthology.orchestra.run.smithi019.stdout: 5 172716 11.88 MB/sec execute 221 sec latency 71.662 ms 2024-03-26T01:59:08.107 INFO:teuthology.orchestra.run.smithi019.stdout:HEALTH_OK 2024-03-26T01:59:08.233 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:07 smithi019 ceph-mon[66359]: from='client.34156 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:08.234 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:07 smithi019 ceph-mon[66359]: pgmap v21: 97 pgs: 97 active+clean; 328 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 5.6 MiB/s wr, 580 op/s 2024-03-26T01:59:08.234 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:07 smithi019 ceph-mon[66359]: Reconfiguring crash.smithi019 (monmap changed)... 2024-03-26T01:59:08.234 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:07 smithi019 ceph-mon[66359]: Reconfiguring daemon crash.smithi019 on smithi019 2024-03-26T01:59:08.234 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:07 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/437434439' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T01:59:08.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:07 smithi069 ceph-mon[53087]: from='client.34156 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:08.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:07 smithi069 ceph-mon[53087]: pgmap v21: 97 pgs: 97 active+clean; 328 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 5.6 MiB/s wr, 580 op/s 2024-03-26T01:59:08.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:07 smithi069 ceph-mon[53087]: Reconfiguring crash.smithi019 (monmap changed)... 2024-03-26T01:59:08.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:07 smithi069 ceph-mon[53087]: Reconfiguring daemon crash.smithi019 on smithi019 2024-03-26T01:59:08.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:07 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/437434439' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T01:59:09.104 INFO:teuthology.orchestra.run.smithi019.stdout: 5 173084 11.87 MB/sec execute 222 sec latency 46.767 ms 2024-03-26T01:59:09.251 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:09 smithi019 ceph-mon[66359]: from='client.44115 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:09.252 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:09 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/808362866' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T01:59:09.252 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:09 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:09.252 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:09 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:09.252 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:09 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T01:59:09.252 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:09 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:09.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:09 smithi069 ceph-mon[53087]: from='client.44115 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:09.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:09 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/808362866' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T01:59:09.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:09 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:09.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:09 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:09.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:09 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T01:59:09.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:09 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:10.022 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:10 smithi019 ceph-mon[66359]: pgmap v22: 97 pgs: 97 active+clean; 326 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 5.0 MiB/s wr, 572 op/s 2024-03-26T01:59:10.022 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:10 smithi019 ceph-mon[66359]: Reconfiguring osd.0 (monmap changed)... 2024-03-26T01:59:10.022 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:10 smithi019 ceph-mon[66359]: Reconfiguring daemon osd.0 on smithi019 2024-03-26T01:59:10.104 INFO:teuthology.orchestra.run.smithi019.stdout: 5 173394 11.87 MB/sec execute 223 sec latency 51.171 ms 2024-03-26T01:59:10.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:10 smithi069 ceph-mon[53087]: pgmap v22: 97 pgs: 97 active+clean; 326 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 5.0 MiB/s wr, 572 op/s 2024-03-26T01:59:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:10 smithi069 ceph-mon[53087]: Reconfiguring osd.0 (monmap changed)... 2024-03-26T01:59:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:10 smithi069 ceph-mon[53087]: Reconfiguring daemon osd.0 on smithi019 2024-03-26T01:59:11.047 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:11 smithi019 ceph-mon[66359]: pgmap v23: 97 pgs: 97 active+clean; 326 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.0 MiB/s wr, 390 op/s 2024-03-26T01:59:11.104 INFO:teuthology.orchestra.run.smithi019.stdout: 5 173656 11.86 MB/sec execute 224 sec latency 47.186 ms 2024-03-26T01:59:11.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:11 smithi069 ceph-mon[53087]: pgmap v23: 97 pgs: 97 active+clean; 326 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.0 MiB/s wr, 390 op/s 2024-03-26T01:59:12.104 INFO:teuthology.orchestra.run.smithi019.stdout: 5 173964 11.85 MB/sec execute 225 sec latency 38.069 ms 2024-03-26T01:59:12.565 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:12 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:12.565 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:12 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:12.565 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:12 smithi019 ceph-mon[66359]: Reconfiguring osd.1 (monmap changed)... 2024-03-26T01:59:12.565 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:12 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T01:59:12.566 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:12 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:12.566 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:12 smithi019 ceph-mon[66359]: Reconfiguring daemon osd.1 on smithi019 2024-03-26T01:59:12.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:12 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:12.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:12 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:12.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:12 smithi069 ceph-mon[53087]: Reconfiguring osd.1 (monmap changed)... 2024-03-26T01:59:12.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:12 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T01:59:12.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:12 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:12.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:12 smithi069 ceph-mon[53087]: Reconfiguring daemon osd.1 on smithi019 2024-03-26T01:59:13.104 INFO:teuthology.orchestra.run.smithi019.stdout: 5 174358 11.84 MB/sec execute 226 sec latency 39.794 ms 2024-03-26T01:59:13.586 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:13 smithi019 ceph-mon[66359]: pgmap v24: 97 pgs: 97 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 6.1 MiB/s wr, 563 op/s 2024-03-26T01:59:13.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:13 smithi069 ceph-mon[53087]: pgmap v24: 97 pgs: 97 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 6.1 MiB/s wr, 563 op/s 2024-03-26T01:59:14.104 INFO:teuthology.orchestra.run.smithi019.stdout: 5 174617 11.81 MB/sec execute 227 sec latency 44.336 ms 2024-03-26T01:59:15.104 INFO:teuthology.orchestra.run.smithi019.stdout: 5 174921 11.80 MB/sec execute 228 sec latency 59.452 ms 2024-03-26T01:59:15.629 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:15 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:15.629 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:15 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:15.629 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:15 smithi019 ceph-mon[66359]: Reconfiguring osd.2 (monmap changed)... 2024-03-26T01:59:15.629 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:15 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T01:59:15.630 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:15 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:15.630 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:15 smithi019 ceph-mon[66359]: pgmap v25: 97 pgs: 97 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.7 MiB/s wr, 370 op/s 2024-03-26T01:59:15.630 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:15 smithi019 ceph-mon[66359]: Reconfiguring daemon osd.2 on smithi019 2024-03-26T01:59:15.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:15 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:15.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:15 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:15.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:15 smithi069 ceph-mon[53087]: Reconfiguring osd.2 (monmap changed)... 2024-03-26T01:59:15.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:15 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T01:59:15.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:15 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:15.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:15 smithi069 ceph-mon[53087]: pgmap v25: 97 pgs: 97 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.7 MiB/s wr, 370 op/s 2024-03-26T01:59:15.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:15 smithi069 ceph-mon[53087]: Reconfiguring daemon osd.2 on smithi019 2024-03-26T01:59:16.104 INFO:teuthology.orchestra.run.smithi019.stdout: 5 175268 11.77 MB/sec execute 229 sec latency 55.599 ms 2024-03-26T01:59:16.641 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:16 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:59:16.858 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:16 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:59:17.104 INFO:teuthology.orchestra.run.smithi019.stdout: 5 175623 11.74 MB/sec execute 230 sec latency 54.146 ms 2024-03-26T01:59:17.696 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:17 smithi019 ceph-mon[66359]: pgmap v26: 97 pgs: 97 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 4.6 MiB/s wr, 474 op/s 2024-03-26T01:59:17.697 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:17 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:17.697 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:17 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:17.697 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:17 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T01:59:17.697 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:17 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:18.034 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:17 smithi069 ceph-mon[53087]: pgmap v26: 97 pgs: 97 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 4.6 MiB/s wr, 474 op/s 2024-03-26T01:59:18.034 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:17 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:18.034 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:17 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:18.034 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:17 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T01:59:18.034 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:17 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:18.104 INFO:teuthology.orchestra.run.smithi019.stdout: 5 175848 11.72 MB/sec execute 231 sec latency 66.350 ms 2024-03-26T01:59:18.761 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:18 smithi019 ceph-mon[66359]: Reconfiguring osd.3 (monmap changed)... 2024-03-26T01:59:18.761 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:18 smithi019 ceph-mon[66359]: Reconfiguring daemon osd.3 on smithi019 2024-03-26T01:59:19.104 INFO:teuthology.orchestra.run.smithi019.stdout: 5 176242 11.73 MB/sec execute 232 sec latency 73.063 ms 2024-03-26T01:59:19.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:18 smithi069 ceph-mon[53087]: Reconfiguring osd.3 (monmap changed)... 2024-03-26T01:59:19.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:18 smithi069 ceph-mon[53087]: Reconfiguring daemon osd.3 on smithi019 2024-03-26T01:59:19.786 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:19 smithi019 ceph-mon[66359]: pgmap v27: 97 pgs: 97 active+clean; 333 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.4 MiB/s wr, 447 op/s 2024-03-26T01:59:20.104 INFO:teuthology.orchestra.run.smithi019.stdout: 5 176619 11.72 MB/sec execute 233 sec latency 64.488 ms 2024-03-26T01:59:20.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:19 smithi069 ceph-mon[53087]: pgmap v27: 97 pgs: 97 active+clean; 333 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.4 MiB/s wr, 447 op/s 2024-03-26T01:59:20.819 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:20 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:20.820 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:20 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:20.820 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:20 smithi019 ceph-mon[66359]: Reconfiguring mds.foofs.smithi019.lmryom (monmap changed)... 2024-03-26T01:59:20.820 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:20 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi019.lmryom", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T01:59:20.820 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:20 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:20.820 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:20 smithi019 ceph-mon[66359]: Reconfiguring daemon mds.foofs.smithi019.lmryom on smithi019 2024-03-26T01:59:21.104 INFO:teuthology.orchestra.run.smithi019.stdout: 5 176944 11.72 MB/sec execute 234 sec latency 49.132 ms 2024-03-26T01:59:21.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:20 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:21.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:20 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:21.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:20 smithi069 ceph-mon[53087]: Reconfiguring mds.foofs.smithi019.lmryom (monmap changed)... 2024-03-26T01:59:21.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:20 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi019.lmryom", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T01:59:21.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:20 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:21.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:20 smithi069 ceph-mon[53087]: Reconfiguring daemon mds.foofs.smithi019.lmryom on smithi019 2024-03-26T01:59:21.726 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:21 smithi019 ceph-mon[66359]: pgmap v28: 97 pgs: 97 active+clean; 333 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.9 MiB/s wr, 381 op/s 2024-03-26T01:59:22.105 INFO:teuthology.orchestra.run.smithi019.stdout: 5 177219 11.72 MB/sec execute 235 sec latency 37.718 ms 2024-03-26T01:59:22.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:21 smithi069 ceph-mon[53087]: pgmap v28: 97 pgs: 97 active+clean; 333 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.9 MiB/s wr, 381 op/s 2024-03-26T01:59:23.106 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:22 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:23.107 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:22 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:23.107 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:22 smithi019 ceph-mon[66359]: Reconfiguring crash.smithi069 (monmap changed)... 2024-03-26T01:59:23.107 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:22 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi069", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-26T01:59:23.107 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:22 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:23.107 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:22 smithi019 ceph-mon[66359]: Reconfiguring daemon crash.smithi069 on smithi069 2024-03-26T01:59:23.107 INFO:teuthology.orchestra.run.smithi019.stdout: 5 177775 11.73 MB/sec execute 236 sec latency 27.246 ms 2024-03-26T01:59:23.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:22 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:23.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:22 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:23.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:22 smithi069 ceph-mon[53087]: Reconfiguring crash.smithi069 (monmap changed)... 2024-03-26T01:59:23.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:22 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi069", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-26T01:59:23.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:22 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:23.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:22 smithi069 ceph-mon[53087]: Reconfiguring daemon crash.smithi069 on smithi069 2024-03-26T01:59:24.075 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:23 smithi019 ceph-mon[66359]: pgmap v29: 97 pgs: 97 active+clean; 333 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 6.0 MiB/s wr, 576 op/s 2024-03-26T01:59:24.075 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:24.075 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:24.075 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi069.wpbewu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T01:59:24.075 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T01:59:24.075 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:24.107 INFO:teuthology.orchestra.run.smithi019.stdout: 5 178180 11.72 MB/sec execute 237 sec latency 26.821 ms 2024-03-26T01:59:24.322 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:23 smithi069 ceph-mon[53087]: pgmap v29: 97 pgs: 97 active+clean; 333 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 6.0 MiB/s wr, 576 op/s 2024-03-26T01:59:24.323 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:24.323 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:24.323 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi069.wpbewu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T01:59:24.323 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T01:59:24.323 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:25.108 INFO:teuthology.orchestra.run.smithi019.stdout: 5 178784 11.72 MB/sec execute 238 sec latency 31.880 ms 2024-03-26T01:59:25.119 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:24 smithi019 ceph-mon[66359]: Reconfiguring mgr.smithi069.wpbewu (monmap changed)... 2024-03-26T01:59:25.119 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:24 smithi019 ceph-mon[66359]: Reconfiguring daemon mgr.smithi069.wpbewu on smithi069 2024-03-26T01:59:25.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:24 smithi069 ceph-mon[53087]: Reconfiguring mgr.smithi069.wpbewu (monmap changed)... 2024-03-26T01:59:25.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:24 smithi069 ceph-mon[53087]: Reconfiguring daemon mgr.smithi069.wpbewu on smithi069 2024-03-26T01:59:25.879 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:25 smithi019 ceph-mon[66359]: pgmap v30: 97 pgs: 97 active+clean; 333 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.9 MiB/s wr, 403 op/s 2024-03-26T01:59:25.879 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:25 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:25.880 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:25 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:25.880 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:25 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:59:25.880 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:25 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T01:59:25.880 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:25 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:26.108 INFO:teuthology.orchestra.run.smithi019.stdout: 5 179247 11.70 MB/sec execute 239 sec latency 30.309 ms 2024-03-26T01:59:26.256 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:25 smithi069 ceph-mon[53087]: pgmap v30: 97 pgs: 97 active+clean; 333 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.9 MiB/s wr, 403 op/s 2024-03-26T01:59:26.256 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:25 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:26.256 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:25 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:26.256 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:25 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:59:26.256 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:25 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T01:59:26.256 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:25 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:27.035 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:26 smithi019 ceph-mon[66359]: Reconfiguring mon.smithi069 (monmap changed)... 2024-03-26T01:59:27.035 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:26 smithi019 ceph-mon[66359]: Reconfiguring daemon mon.smithi069 on smithi069 2024-03-26T01:59:27.108 INFO:teuthology.orchestra.run.smithi019.stdout: 5 179826 11.75 MB/sec execute 240 sec latency 44.678 ms 2024-03-26T01:59:27.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:26 smithi069 ceph-mon[53087]: Reconfiguring mon.smithi069 (monmap changed)... 2024-03-26T01:59:27.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:26 smithi069 ceph-mon[53087]: Reconfiguring daemon mon.smithi069 on smithi069 2024-03-26T01:59:28.050 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:27 smithi019 ceph-mon[66359]: pgmap v31: 97 pgs: 97 active+clean; 339 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 6.5 MiB/s wr, 679 op/s 2024-03-26T01:59:28.050 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:27 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:28.050 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:27 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:28.108 INFO:teuthology.orchestra.run.smithi019.stdout: 5 180409 11.77 MB/sec execute 241 sec latency 65.274 ms 2024-03-26T01:59:28.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:27 smithi069 ceph-mon[53087]: pgmap v31: 97 pgs: 97 active+clean; 339 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 6.5 MiB/s wr, 679 op/s 2024-03-26T01:59:28.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:27 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:28.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:27 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:29.056 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:28 smithi019 ceph-mon[66359]: Reconfiguring osd.4 (monmap changed)... 2024-03-26T01:59:29.057 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:28 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T01:59:29.057 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:28 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:29.057 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:28 smithi019 ceph-mon[66359]: Reconfiguring daemon osd.4 on smithi069 2024-03-26T01:59:29.108 INFO:teuthology.orchestra.run.smithi019.stdout: 5 180813 11.78 MB/sec execute 242 sec latency 28.324 ms 2024-03-26T01:59:29.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:28 smithi069 ceph-mon[53087]: Reconfiguring osd.4 (monmap changed)... 2024-03-26T01:59:29.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:28 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T01:59:29.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:28 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:29.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:28 smithi069 ceph-mon[53087]: Reconfiguring daemon osd.4 on smithi069 2024-03-26T01:59:30.062 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:29 smithi019 ceph-mon[66359]: pgmap v32: 97 pgs: 97 active+clean; 339 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 6.0 MiB/s wr, 601 op/s 2024-03-26T01:59:30.063 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:29 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:30.063 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:29 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:30.063 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:29 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T01:59:30.063 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:29 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:30.108 INFO:teuthology.orchestra.run.smithi019.stdout: 5 181337 11.78 MB/sec execute 243 sec latency 24.217 ms 2024-03-26T01:59:30.300 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:29 smithi069 ceph-mon[53087]: pgmap v32: 97 pgs: 97 active+clean; 339 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 6.0 MiB/s wr, 601 op/s 2024-03-26T01:59:30.301 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:29 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:30.301 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:29 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:30.301 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:29 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T01:59:30.301 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:29 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:31.081 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:30 smithi019 ceph-mon[66359]: Reconfiguring osd.5 (monmap changed)... 2024-03-26T01:59:31.081 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:30 smithi019 ceph-mon[66359]: Reconfiguring daemon osd.5 on smithi069 2024-03-26T01:59:31.082 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:59:31.108 INFO:teuthology.orchestra.run.smithi019.stdout: 5 181705 11.78 MB/sec execute 244 sec latency 38.884 ms 2024-03-26T01:59:31.320 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:30 smithi069 ceph-mon[53087]: Reconfiguring osd.5 (monmap changed)... 2024-03-26T01:59:31.320 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:30 smithi069 ceph-mon[53087]: Reconfiguring daemon osd.5 on smithi069 2024-03-26T01:59:31.321 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:59:32.092 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:31 smithi019 ceph-mon[66359]: pgmap v33: 97 pgs: 97 active+clean; 339 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 5.0 MiB/s wr, 497 op/s 2024-03-26T01:59:32.092 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:31 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:32.093 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:31 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:32.093 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:31 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T01:59:32.093 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:31 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:32.108 INFO:teuthology.orchestra.run.smithi019.stdout: 5 182188 11.77 MB/sec execute 245 sec latency 27.325 ms 2024-03-26T01:59:32.206 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:31 smithi069 ceph-mon[53087]: pgmap v33: 97 pgs: 97 active+clean; 339 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 5.0 MiB/s wr, 497 op/s 2024-03-26T01:59:32.207 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:31 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:32.207 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:31 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:32.207 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:31 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T01:59:32.207 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:31 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:33.096 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:32 smithi019 ceph-mon[66359]: Reconfiguring osd.6 (monmap changed)... 2024-03-26T01:59:33.096 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:32 smithi019 ceph-mon[66359]: Reconfiguring daemon osd.6 on smithi069 2024-03-26T01:59:33.108 INFO:teuthology.orchestra.run.smithi019.stdout: 5 182722 11.75 MB/sec execute 246 sec latency 54.044 ms 2024-03-26T01:59:33.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:32 smithi069 ceph-mon[53087]: Reconfiguring osd.6 (monmap changed)... 2024-03-26T01:59:33.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:32 smithi069 ceph-mon[53087]: Reconfiguring daemon osd.6 on smithi069 2024-03-26T01:59:34.101 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:33 smithi019 ceph-mon[66359]: pgmap v34: 97 pgs: 97 active+clean; 341 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 8.3 MiB/s wr, 789 op/s 2024-03-26T01:59:34.102 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:33 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:34.102 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:33 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:34.102 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:33 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T01:59:34.102 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:33 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:34.108 INFO:teuthology.orchestra.run.smithi019.stdout: 5 183393 11.81 MB/sec execute 247 sec latency 29.020 ms 2024-03-26T01:59:34.166 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:33 smithi069 ceph-mon[53087]: pgmap v34: 97 pgs: 97 active+clean; 341 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 8.3 MiB/s wr, 789 op/s 2024-03-26T01:59:34.167 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:33 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:34.167 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:33 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:34.167 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:33 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T01:59:34.167 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:33 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:35.107 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:34 smithi019 ceph-mon[66359]: Reconfiguring osd.7 (monmap changed)... 2024-03-26T01:59:35.108 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:34 smithi019 ceph-mon[66359]: Reconfiguring daemon osd.7 on smithi069 2024-03-26T01:59:35.108 INFO:teuthology.orchestra.run.smithi019.stdout: 5 184090 11.84 MB/sec execute 248 sec latency 35.586 ms 2024-03-26T01:59:35.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:34 smithi069 ceph-mon[53087]: Reconfiguring osd.7 (monmap changed)... 2024-03-26T01:59:35.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:34 smithi069 ceph-mon[53087]: Reconfiguring daemon osd.7 on smithi069 2024-03-26T01:59:36.022 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:35 smithi019 ceph-mon[66359]: pgmap v35: 97 pgs: 97 active+clean; 341 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 7.8 KiB/s rd, 6.2 MiB/s wr, 594 op/s 2024-03-26T01:59:36.022 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:35 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:36.023 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:35 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:36.023 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:35 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi069.spgwgo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T01:59:36.023 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:35 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:36.108 INFO:teuthology.orchestra.run.smithi019.stdout: 5 184734 11.86 MB/sec execute 249 sec latency 31.290 ms 2024-03-26T01:59:36.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:35 smithi069 ceph-mon[53087]: pgmap v35: 97 pgs: 97 active+clean; 341 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 7.8 KiB/s rd, 6.2 MiB/s wr, 594 op/s 2024-03-26T01:59:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:35 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:35 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:35 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi069.spgwgo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T01:59:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:35 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:37.108 INFO:teuthology.orchestra.run.smithi019.stdout: 5 185088 11.86 MB/sec execute 250 sec latency 48.782 ms 2024-03-26T01:59:37.108 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:36 smithi069 ceph-mon[53087]: Reconfiguring mds.foofs.smithi069.spgwgo (monmap changed)... 2024-03-26T01:59:37.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:36 smithi069 ceph-mon[53087]: Reconfiguring daemon mds.foofs.smithi069.spgwgo on smithi069 2024-03-26T01:59:37.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:36 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:37.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:36 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:37.138 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:36 smithi019 ceph-mon[66359]: Reconfiguring mds.foofs.smithi069.spgwgo (monmap changed)... 2024-03-26T01:59:37.138 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:36 smithi019 ceph-mon[66359]: Reconfiguring daemon mds.foofs.smithi069.spgwgo on smithi069 2024-03-26T01:59:37.139 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:36 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:37.139 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:36 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:38.108 INFO:teuthology.orchestra.run.smithi019.stdout: 5 185297 11.83 MB/sec execute 251 sec latency 47.756 ms 2024-03-26T01:59:38.159 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:37 smithi019 ceph-mon[66359]: pgmap v36: 97 pgs: 97 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.8 KiB/s rd, 9.1 MiB/s wr, 875 op/s 2024-03-26T01:59:38.160 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:37 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:59:38.160 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:37 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T01:59:38.160 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:37 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T01:59:38.160 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:37 smithi019 ceph-mon[66359]: Upgrade: Setting container_image for all mon 2024-03-26T01:59:38.160 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:37 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:38.160 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:37 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi019"}]: dispatch 2024-03-26T01:59:38.160 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:37 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi019"}]': finished 2024-03-26T01:59:38.160 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:37 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi069"}]: dispatch 2024-03-26T01:59:38.160 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:37 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi069"}]': finished 2024-03-26T01:59:38.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:37 smithi069 ceph-mon[53087]: pgmap v36: 97 pgs: 97 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.8 KiB/s rd, 9.1 MiB/s wr, 875 op/s 2024-03-26T01:59:38.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:37 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:59:38.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:37 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T01:59:38.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:37 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T01:59:38.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:37 smithi069 ceph-mon[53087]: Upgrade: Setting container_image for all mon 2024-03-26T01:59:38.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:37 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:38.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:37 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi019"}]: dispatch 2024-03-26T01:59:38.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:37 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi019"}]': finished 2024-03-26T01:59:38.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:37 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi069"}]: dispatch 2024-03-26T01:59:38.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:37 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi069"}]': finished 2024-03-26T01:59:38.685 INFO:teuthology.orchestra.run.smithi019.stdout:true 2024-03-26T01:59:39.108 INFO:teuthology.orchestra.run.smithi019.stdout: 5 185540 11.82 MB/sec execute 252 sec latency 98.050 ms 2024-03-26T01:59:39.682 INFO:teuthology.orchestra.run.smithi019.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T01:59:39.682 INFO:teuthology.orchestra.run.smithi019.stdout:alertmanager.smithi019 smithi019 *:9093,9094 running (3m) 58s ago 12m 16.2M - 0.23.0 ba2b418f427c a6630841c2b2 2024-03-26T01:59:39.682 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi019 smithi019 running (12m) 58s ago 12m 6405k - 17.2.0 e1d6a67b021e 713db693d5c1 2024-03-26T01:59:39.682 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi069 smithi069 running (12m) 43s ago 12m 6405k - 17.2.0 e1d6a67b021e a87d27e2b74b 2024-03-26T01:59:39.682 INFO:teuthology.orchestra.run.smithi019.stdout:grafana.smithi019 smithi019 *:3000 running (3m) 58s ago 12m 45.4M - 8.3.5 dad864ee21e9 a97dea3b5fec 2024-03-26T01:59:39.682 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi019.lmryom smithi019 running (6m) 58s ago 6m 37.8M - 17.2.0 e1d6a67b021e 91a7c2942715 2024-03-26T01:59:39.682 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi069.spgwgo smithi069 running (6m) 43s ago 6m 14.2M - 17.2.0 e1d6a67b021e 94d96338dbe1 2024-03-26T01:59:39.682 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi019.ixuvdm smithi019 *:8443,9283,8765 running (2m) 58s ago 14m 478M - 19.0.0-1775-g5e70124b 5f21c9264136 4460b22c7a56 2024-03-26T01:59:39.683 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi069.wpbewu smithi069 *:8443,9283,8765 running (99s) 43s ago 12m 450M - 19.0.0-1775-g5e70124b 5f21c9264136 380fd33842a8 2024-03-26T01:59:39.683 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi019 smithi019 running (81s) 58s ago 14m 33.6M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 c18934705774 2024-03-26T01:59:39.683 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi069 smithi069 running (49s) 43s ago 12m 24.3M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 a856bd263104 2024-03-26T01:59:39.683 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.0.0.smithi019.qbizxo smithi019 *:2049 running (6m) 58s ago 6m 286M - 4.0 e1d6a67b021e cd6628a9075d 2024-03-26T01:59:39.683 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.1.0.smithi069.uawdhc smithi069 *:2049 running (6m) 43s ago 6m 85.3M - 4.0 e1d6a67b021e 6ee3695e7a10 2024-03-26T01:59:39.683 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi019 smithi019 *:9100 running (3m) 58s ago 12m 13.1M - 1.3.1 1dbe0e931976 0d1d184bec63 2024-03-26T01:59:39.683 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi069 smithi069 *:9100 running (2m) 43s ago 11m 20.9M - 1.3.1 1dbe0e931976 c49a006f94e7 2024-03-26T01:59:39.683 INFO:teuthology.orchestra.run.smithi019.stdout:osd.0 smithi019 running (11m) 58s ago 11m 557M 1937M 17.2.0 e1d6a67b021e c1a797230ab6 2024-03-26T01:59:39.683 INFO:teuthology.orchestra.run.smithi019.stdout:osd.1 smithi019 running (10m) 58s ago 10m 476M 1937M 17.2.0 e1d6a67b021e 31d3489c7188 2024-03-26T01:59:39.683 INFO:teuthology.orchestra.run.smithi019.stdout:osd.2 smithi019 running (10m) 58s ago 10m 355M 1937M 17.2.0 e1d6a67b021e f549115e7685 2024-03-26T01:59:39.683 INFO:teuthology.orchestra.run.smithi019.stdout:osd.3 smithi019 running (9m) 58s ago 9m 489M 1937M 17.2.0 e1d6a67b021e 154af68d03e1 2024-03-26T01:59:39.683 INFO:teuthology.orchestra.run.smithi019.stdout:osd.4 smithi069 running (9m) 43s ago 9m 529M 2704M 17.2.0 e1d6a67b021e 4fd5e853f8bf 2024-03-26T01:59:39.683 INFO:teuthology.orchestra.run.smithi019.stdout:osd.5 smithi069 running (9m) 43s ago 9m 360M 2704M 17.2.0 e1d6a67b021e 932e401d87c9 2024-03-26T01:59:39.683 INFO:teuthology.orchestra.run.smithi019.stdout:osd.6 smithi069 running (8m) 43s ago 8m 546M 2704M 17.2.0 e1d6a67b021e b123f75154c8 2024-03-26T01:59:39.684 INFO:teuthology.orchestra.run.smithi019.stdout:osd.7 smithi069 running (8m) 43s ago 8m 456M 2704M 17.2.0 e1d6a67b021e d175fbc9ff8a 2024-03-26T01:59:39.684 INFO:teuthology.orchestra.run.smithi019.stdout:prometheus.smithi019 smithi019 *:9095 running (106s) 58s ago 11m 52.6M - 2.33.4 514e6a882f6e 29f89db72a07 2024-03-26T01:59:39.991 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:39 smithi019 ceph-mon[66359]: pgmap v37: 97 pgs: 97 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 7.2 MiB/s wr, 676 op/s 2024-03-26T01:59:39.991 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:39 smithi019 ceph-mon[66359]: from='client.34170 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:40.108 INFO:teuthology.orchestra.run.smithi019.stdout: 5 185837 11.78 MB/sec execute 253 sec latency 61.203 ms 2024-03-26T01:59:40.305 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T01:59:40.305 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": { 2024-03-26T01:59:40.305 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T01:59:40.305 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:59:40.305 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": { 2024-03-26T01:59:40.305 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T01:59:40.305 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:59:40.306 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": { 2024-03-26T01:59:40.306 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T01:59:40.306 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:59:40.306 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": { 2024-03-26T01:59:40.306 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T01:59:40.306 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T01:59:40.306 INFO:teuthology.orchestra.run.smithi019.stdout: "overall": { 2024-03-26T01:59:40.306 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-03-26T01:59:40.306 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 4 2024-03-26T01:59:40.306 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-26T01:59:40.306 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T01:59:40.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:39 smithi069 ceph-mon[53087]: pgmap v37: 97 pgs: 97 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 7.2 MiB/s wr, 676 op/s 2024-03-26T01:59:40.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:39 smithi069 ceph-mon[53087]: from='client.34170 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:40.870 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T01:59:40.870 INFO:teuthology.orchestra.run.smithi019.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2", 2024-03-26T01:59:40.870 INFO:teuthology.orchestra.run.smithi019.stdout: "in_progress": true, 2024-03-26T01:59:40.870 INFO:teuthology.orchestra.run.smithi019.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T01:59:40.870 INFO:teuthology.orchestra.run.smithi019.stdout: "services_complete": [ 2024-03-26T01:59:40.870 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr", 2024-03-26T01:59:40.870 INFO:teuthology.orchestra.run.smithi019.stdout: "mon" 2024-03-26T01:59:40.870 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-26T01:59:40.870 INFO:teuthology.orchestra.run.smithi019.stdout: "progress": "4/23 daemons upgraded", 2024-03-26T01:59:40.870 INFO:teuthology.orchestra.run.smithi019.stdout: "message": "Currently upgrading crash daemons", 2024-03-26T01:59:40.870 INFO:teuthology.orchestra.run.smithi019.stdout: "is_paused": false 2024-03-26T01:59:40.871 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T01:59:41.008 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:40 smithi019 ceph-mon[66359]: from='client.34174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:41.008 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:40 smithi019 ceph-mon[66359]: from='client.44129 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:41.009 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:40 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/2414989830' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T01:59:41.009 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:41.009 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi019", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-26T01:59:41.009 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:41.108 INFO:teuthology.orchestra.run.smithi019.stdout: 5 186239 11.76 MB/sec execute 254 sec latency 37.420 ms 2024-03-26T01:59:41.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:40 smithi069 ceph-mon[53087]: from='client.34174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:41.383 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:40 smithi069 ceph-mon[53087]: from='client.44129 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:41.384 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:40 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/2414989830' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T01:59:41.384 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:41.384 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi019", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-26T01:59:41.384 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:41.504 INFO:teuthology.orchestra.run.smithi019.stdout:HEALTH_OK 2024-03-26T01:59:42.018 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:41 smithi019 ceph-mon[66359]: Upgrade: Updating crash.smithi019 (1/2) 2024-03-26T01:59:42.018 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:41 smithi019 ceph-mon[66359]: Deploying daemon crash.smithi019 on smithi019 2024-03-26T01:59:42.019 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:41 smithi019 ceph-mon[66359]: pgmap v38: 97 pgs: 97 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 6.9 MiB/s wr, 650 op/s 2024-03-26T01:59:42.019 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:41 smithi019 ceph-mon[66359]: from='client.34182 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:42.019 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:41 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/59860191' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T01:59:42.108 INFO:teuthology.orchestra.run.smithi019.stdout: 5 186770 11.79 MB/sec execute 255 sec latency 34.634 ms 2024-03-26T01:59:42.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:41 smithi069 ceph-mon[53087]: Upgrade: Updating crash.smithi019 (1/2) 2024-03-26T01:59:42.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:41 smithi069 ceph-mon[53087]: Deploying daemon crash.smithi019 on smithi019 2024-03-26T01:59:42.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:41 smithi069 ceph-mon[53087]: pgmap v38: 97 pgs: 97 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 6.9 MiB/s wr, 650 op/s 2024-03-26T01:59:42.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:41 smithi069 ceph-mon[53087]: from='client.34182 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:42.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:41 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/59860191' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T01:59:43.108 INFO:teuthology.orchestra.run.smithi019.stdout: 5 187368 11.82 MB/sec execute 256 sec latency 33.326 ms 2024-03-26T01:59:44.045 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:43 smithi019 ceph-mon[66359]: pgmap v39: 97 pgs: 97 active+clean; 337 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 13 KiB/s rd, 8.6 MiB/s wr, 848 op/s 2024-03-26T01:59:44.108 INFO:teuthology.orchestra.run.smithi019.stdout: 5 187653 11.80 MB/sec execute 257 sec latency 60.002 ms 2024-03-26T01:59:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:43 smithi069 ceph-mon[53087]: pgmap v39: 97 pgs: 97 active+clean; 337 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 13 KiB/s rd, 8.6 MiB/s wr, 848 op/s 2024-03-26T01:59:45.108 INFO:teuthology.orchestra.run.smithi019.stdout: 5 187935 11.79 MB/sec execute 258 sec latency 36.115 ms 2024-03-26T01:59:46.070 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:45 smithi019 ceph-mon[66359]: pgmap v40: 97 pgs: 97 active+clean; 337 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.8 KiB/s rd, 5.4 MiB/s wr, 556 op/s 2024-03-26T01:59:46.071 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:59:46.108 INFO:teuthology.orchestra.run.smithi019.stdout: 5 188278 11.77 MB/sec execute 259 sec latency 56.704 ms 2024-03-26T01:59:46.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:45 smithi069 ceph-mon[53087]: pgmap v40: 97 pgs: 97 active+clean; 337 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.8 KiB/s rd, 5.4 MiB/s wr, 556 op/s 2024-03-26T01:59:46.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T01:59:47.108 INFO:teuthology.orchestra.run.smithi019.stdout: 5 188905 11.80 MB/sec execute 260 sec latency 30.762 ms 2024-03-26T01:59:48.083 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:47 smithi019 ceph-mon[66359]: pgmap v41: 97 pgs: 97 active+clean; 336 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.8 KiB/s rd, 7.3 MiB/s wr, 711 op/s 2024-03-26T01:59:48.083 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:47 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:48.084 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:47 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:48.108 INFO:teuthology.orchestra.run.smithi019.stdout: 5 189558 11.79 MB/sec execute 261 sec latency 30.618 ms 2024-03-26T01:59:48.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:47 smithi069 ceph-mon[53087]: pgmap v41: 97 pgs: 97 active+clean; 336 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.8 KiB/s rd, 7.3 MiB/s wr, 711 op/s 2024-03-26T01:59:48.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:47 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:48.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:47 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:49.108 INFO:teuthology.orchestra.run.smithi019.stdout: 5 190097 11.81 MB/sec execute 262 sec latency 35.651 ms 2024-03-26T01:59:50.108 INFO:teuthology.orchestra.run.smithi019.stdout: 5 190758 11.84 MB/sec execute 263 sec latency 29.184 ms 2024-03-26T01:59:50.110 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:49 smithi019 ceph-mon[66359]: pgmap v42: 97 pgs: 97 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 5.4 MiB/s wr, 527 op/s 2024-03-26T01:59:50.110 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:49 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:50.110 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:49 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi069", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-26T01:59:50.110 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:49 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:50.292 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:49 smithi069 ceph-mon[53087]: pgmap v42: 97 pgs: 97 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 5.4 MiB/s wr, 527 op/s 2024-03-26T01:59:50.292 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:49 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:50.292 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:49 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi069", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-26T01:59:50.292 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:49 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:51.108 INFO:teuthology.orchestra.run.smithi019.stdout: 5 191229 11.86 MB/sec execute 264 sec latency 42.141 ms 2024-03-26T01:59:51.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:50 smithi069 ceph-mon[53087]: Upgrade: Updating crash.smithi069 (2/2) 2024-03-26T01:59:51.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:50 smithi069 ceph-mon[53087]: Deploying daemon crash.smithi069 on smithi069 2024-03-26T01:59:51.116 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:50 smithi019 ceph-mon[66359]: Upgrade: Updating crash.smithi069 (2/2) 2024-03-26T01:59:51.116 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:50 smithi019 ceph-mon[66359]: Deploying daemon crash.smithi069 on smithi069 2024-03-26T01:59:52.089 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:51 smithi069 ceph-mon[53087]: pgmap v43: 97 pgs: 97 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.6 MiB/s wr, 451 op/s 2024-03-26T01:59:52.108 INFO:teuthology.orchestra.run.smithi019.stdout: 5 191676 11.86 MB/sec execute 265 sec latency 37.410 ms 2024-03-26T01:59:52.126 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:51 smithi019 ceph-mon[66359]: pgmap v43: 97 pgs: 97 active+clean; 344 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.6 MiB/s wr, 451 op/s 2024-03-26T01:59:53.108 INFO:teuthology.orchestra.run.smithi019.stdout: 5 191941 11.84 MB/sec execute 266 sec latency 47.335 ms 2024-03-26T01:59:54.108 INFO:teuthology.orchestra.run.smithi019.stdout: 5 192145 11.81 MB/sec execute 267 sec latency 66.617 ms 2024-03-26T01:59:54.206 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:53 smithi019 ceph-mon[66359]: pgmap v44: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 8.0 MiB/s wr, 790 op/s 2024-03-26T01:59:54.206 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:53 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:54.206 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:53 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:54.207 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:53 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:59:54.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:53 smithi069 ceph-mon[53087]: pgmap v44: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 8.0 MiB/s wr, 790 op/s 2024-03-26T01:59:54.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:53 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:54.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:53 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:54.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:53 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:59:55.109 INFO:teuthology.orchestra.run.smithi019.stdout: 5 192353 11.79 MB/sec execute 268 sec latency 60.081 ms 2024-03-26T01:59:56.082 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:55 smithi019 ceph-mon[66359]: pgmap v45: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 6.3 MiB/s wr, 592 op/s 2024-03-26T01:59:56.108 INFO:teuthology.orchestra.run.smithi019.stdout: 5 192606 11.78 MB/sec execute 269 sec latency 61.680 ms 2024-03-26T01:59:56.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:55 smithi069 ceph-mon[53087]: pgmap v45: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 6.3 MiB/s wr, 592 op/s 2024-03-26T01:59:57.109 INFO:teuthology.orchestra.run.smithi019.stdout: 5 192939 11.75 MB/sec execute 270 sec latency 53.154 ms 2024-03-26T01:59:58.109 INFO:teuthology.orchestra.run.smithi019.stdout: 5 193289 11.72 MB/sec execute 271 sec latency 54.199 ms 2024-03-26T01:59:58.119 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:57 smithi019 ceph-mon[66359]: pgmap v46: 97 pgs: 97 active+clean; 338 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 7.6 MiB/s wr, 742 op/s 2024-03-26T01:59:58.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:57 smithi069 ceph-mon[53087]: pgmap v46: 97 pgs: 97 active+clean; 338 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 7.6 MiB/s wr, 742 op/s 2024-03-26T01:59:59.109 INFO:teuthology.orchestra.run.smithi019.stdout: 5 193478 11.70 MB/sec execute 272 sec latency 40.152 ms 2024-03-26T01:59:59.131 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:58 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:59.131 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:58 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:59.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:58 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T01:59:59.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:58 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:00.068 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:59 smithi069 ceph-mon[53087]: pgmap v47: 97 pgs: 97 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 5.9 MiB/s wr, 608 op/s 2024-03-26T02:00:00.068 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:59 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:00.068 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 01:59:59 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:00.109 INFO:teuthology.orchestra.run.smithi019.stdout: 5 193833 11.72 MB/sec execute 273 sec latency 59.733 ms 2024-03-26T02:00:00.148 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:59 smithi019 ceph-mon[66359]: pgmap v47: 97 pgs: 97 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 5.9 MiB/s wr, 608 op/s 2024-03-26T02:00:00.148 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:59 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:00.148 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 01:59:59 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:01.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:00 smithi069 ceph-mon[53087]: overall HEALTH_OK 2024-03-26T02:00:01.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:00 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:00:01.109 INFO:teuthology.orchestra.run.smithi019.stdout: 5 194312 11.71 MB/sec execute 274 sec latency 42.381 ms 2024-03-26T02:00:01.155 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:00 smithi019 ceph-mon[66359]: overall HEALTH_OK 2024-03-26T02:00:01.155 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:00 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:00:02.109 INFO:teuthology.orchestra.run.smithi019.stdout: 5 194706 11.72 MB/sec execute 275 sec latency 53.646 ms 2024-03-26T02:00:02.184 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:01 smithi019 ceph-mon[66359]: pgmap v48: 97 pgs: 97 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 4.9 MiB/s wr, 511 op/s 2024-03-26T02:00:02.303 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:01 smithi069 ceph-mon[53087]: pgmap v48: 97 pgs: 97 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 4.9 MiB/s wr, 511 op/s 2024-03-26T02:00:03.109 INFO:teuthology.orchestra.run.smithi019.stdout: 5 195004 11.71 MB/sec execute 276 sec latency 33.443 ms 2024-03-26T02:00:03.200 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:02 smithi019 ceph-mon[66359]: mgrmap e41: smithi019.ixuvdm(active, since 91s), standbys: smithi069.wpbewu 2024-03-26T02:00:03.200 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:02 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:03.201 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:02 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:03.201 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:02 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:03.201 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:02 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:03.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:02 smithi069 ceph-mon[53087]: mgrmap e41: smithi019.ixuvdm(active, since 91s), standbys: smithi069.wpbewu 2024-03-26T02:00:03.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:02 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:03.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:02 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:03.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:02 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:03.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:02 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:04.109 INFO:teuthology.orchestra.run.smithi019.stdout: 5 195327 11.69 MB/sec execute 277 sec latency 90.306 ms 2024-03-26T02:00:04.222 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:03 smithi019 ceph-mon[66359]: pgmap v49: 97 pgs: 97 active+clean; 338 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.0 MiB/s wr, 718 op/s 2024-03-26T02:00:04.222 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:03 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:04.222 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:03 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:04.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:03 smithi069 ceph-mon[53087]: pgmap v49: 97 pgs: 97 active+clean; 338 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.0 MiB/s wr, 718 op/s 2024-03-26T02:00:04.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:03 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:04.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:03 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:05.109 INFO:teuthology.orchestra.run.smithi019.stdout: 5 195623 11.68 MB/sec execute 278 sec latency 37.875 ms 2024-03-26T02:00:06.021 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:05 smithi019 ceph-mon[66359]: pgmap v50: 97 pgs: 97 active+clean; 338 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.6 MiB/s wr, 378 op/s 2024-03-26T02:00:06.109 INFO:teuthology.orchestra.run.smithi019.stdout: 5 195884 11.67 MB/sec execute 279 sec latency 51.957 ms 2024-03-26T02:00:06.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:05 smithi069 ceph-mon[53087]: pgmap v50: 97 pgs: 97 active+clean; 338 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.6 MiB/s wr, 378 op/s 2024-03-26T02:00:07.109 INFO:teuthology.orchestra.run.smithi019.stdout: 5 196243 11.66 MB/sec execute 280 sec latency 64.471 ms 2024-03-26T02:00:08.079 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:08 smithi019 ceph-mon[66359]: pgmap v51: 97 pgs: 97 active+clean; 336 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 5.2 MiB/s wr, 516 op/s 2024-03-26T02:00:08.110 INFO:teuthology.orchestra.run.smithi019.stdout: 5 196680 11.63 MB/sec execute 281 sec latency 46.675 ms 2024-03-26T02:00:08.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:08 smithi069 ceph-mon[53087]: pgmap v51: 97 pgs: 97 active+clean; 336 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 5.2 MiB/s wr, 516 op/s 2024-03-26T02:00:09.109 INFO:teuthology.orchestra.run.smithi019.stdout: 5 196956 11.61 MB/sec execute 282 sec latency 33.524 ms 2024-03-26T02:00:10.109 INFO:teuthology.orchestra.run.smithi019.stdout: 5 197375 11.64 MB/sec execute 283 sec latency 59.525 ms 2024-03-26T02:00:10.119 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:09 smithi019 ceph-mon[66359]: pgmap v52: 97 pgs: 97 active+clean; 335 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.7 KiB/s rd, 4.4 MiB/s wr, 428 op/s 2024-03-26T02:00:10.120 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:09 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:10.120 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:09 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:10.120 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:09 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:10.120 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:09 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:00:10.120 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:09 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:10.120 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:09 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:10.120 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:09 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:00:10.120 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:09 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:00:10.120 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:09 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:00:10.120 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:09 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:00:10.120 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:09 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:10.120 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:09 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi019"}]: dispatch 2024-03-26T02:00:10.121 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:09 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi019"}]': finished 2024-03-26T02:00:10.121 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:09 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi069"}]: dispatch 2024-03-26T02:00:10.121 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:09 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi069"}]': finished 2024-03-26T02:00:10.121 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:09 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-26T02:00:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:09 smithi069 ceph-mon[53087]: pgmap v52: 97 pgs: 97 active+clean; 335 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.7 KiB/s rd, 4.4 MiB/s wr, 428 op/s 2024-03-26T02:00:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:09 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:09 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:09 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:09 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:00:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:09 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:09 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:09 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:00:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:09 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:00:10.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:09 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:00:10.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:09 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:00:10.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:09 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:10.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:09 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi019"}]: dispatch 2024-03-26T02:00:10.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:09 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi019"}]': finished 2024-03-26T02:00:10.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:09 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi069"}]: dispatch 2024-03-26T02:00:10.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:09 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi069"}]': finished 2024-03-26T02:00:10.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:09 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-26T02:00:11.110 INFO:teuthology.orchestra.run.smithi019.stdout: 5 197868 11.64 MB/sec execute 284 sec latency 43.758 ms 2024-03-26T02:00:11.139 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:10 smithi019 ceph-mon[66359]: Upgrade: Setting container_image for all crash 2024-03-26T02:00:11.140 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:10 smithi019 ceph-mon[66359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-26T02:00:11.140 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:10 smithi019 ceph-mon[66359]: Upgrade: osd.0 is safe to restart 2024-03-26T02:00:11.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:10 smithi069 ceph-mon[53087]: Upgrade: Setting container_image for all crash 2024-03-26T02:00:11.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:10 smithi069 ceph-mon[53087]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-26T02:00:11.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:10 smithi069 ceph-mon[53087]: Upgrade: osd.0 is safe to restart 2024-03-26T02:00:12.096 INFO:teuthology.orchestra.run.smithi019.stdout:true 2024-03-26T02:00:12.109 INFO:teuthology.orchestra.run.smithi019.stdout: 5 198216 11.65 MB/sec execute 285 sec latency 40.828 ms 2024-03-26T02:00:12.144 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:11 smithi019 ceph-mon[66359]: pgmap v53: 97 pgs: 97 active+clean; 335 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.2 MiB/s wr, 406 op/s 2024-03-26T02:00:12.145 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:11 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:12.145 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:11 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T02:00:12.145 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:11 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:12.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:11 smithi069 ceph-mon[53087]: pgmap v53: 97 pgs: 97 active+clean; 335 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.2 MiB/s wr, 406 op/s 2024-03-26T02:00:12.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:11 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:12.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:11 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T02:00:12.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:11 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:13.110 INFO:teuthology.orchestra.run.smithi019.stdout: 5 198520 11.63 MB/sec execute 286 sec latency 52.523 ms 2024-03-26T02:00:13.137 INFO:teuthology.orchestra.run.smithi019.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:00:13.137 INFO:teuthology.orchestra.run.smithi019.stdout:alertmanager.smithi019 smithi019 *:9093,9094 running (4m) 10s ago 13m 16.2M - 0.23.0 ba2b418f427c a6630841c2b2 2024-03-26T02:00:13.138 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi019 smithi019 running (26s) 10s ago 13m 6660k - 19.0.0-1775-g5e70124b 5f21c9264136 c5f91a47ddf2 2024-03-26T02:00:13.138 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi069 smithi069 running (19s) 15s ago 12m 6669k - 19.0.0-1775-g5e70124b 5f21c9264136 05128803b49e 2024-03-26T02:00:13.138 INFO:teuthology.orchestra.run.smithi019.stdout:grafana.smithi019 smithi019 *:3000 running (3m) 10s ago 13m 46.9M - 8.3.5 dad864ee21e9 a97dea3b5fec 2024-03-26T02:00:13.138 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi019.lmryom smithi019 running (7m) 10s ago 7m 38.4M - 17.2.0 e1d6a67b021e 91a7c2942715 2024-03-26T02:00:13.138 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi069.spgwgo smithi069 running (7m) 15s ago 7m 14.4M - 17.2.0 e1d6a67b021e 94d96338dbe1 2024-03-26T02:00:13.138 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi019.ixuvdm smithi019 *:8443,9283,8765 running (3m) 10s ago 14m 497M - 19.0.0-1775-g5e70124b 5f21c9264136 4460b22c7a56 2024-03-26T02:00:13.138 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi069.wpbewu smithi069 *:8443,9283,8765 running (2m) 15s ago 12m 451M - 19.0.0-1775-g5e70124b 5f21c9264136 380fd33842a8 2024-03-26T02:00:13.138 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi019 smithi019 running (114s) 10s ago 14m 44.9M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 c18934705774 2024-03-26T02:00:13.138 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi069 smithi069 running (83s) 15s ago 12m 35.6M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 a856bd263104 2024-03-26T02:00:13.138 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.0.0.smithi019.qbizxo smithi019 *:2049 running (6m) 10s ago 6m 281M - 4.0 e1d6a67b021e cd6628a9075d 2024-03-26T02:00:13.138 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.1.0.smithi069.uawdhc smithi069 *:2049 running (6m) 15s ago 6m 85.3M - 4.0 e1d6a67b021e 6ee3695e7a10 2024-03-26T02:00:13.138 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi019 smithi019 *:9100 running (3m) 10s ago 13m 11.2M - 1.3.1 1dbe0e931976 0d1d184bec63 2024-03-26T02:00:13.138 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi069 smithi069 *:9100 running (3m) 15s ago 12m 19.2M - 1.3.1 1dbe0e931976 c49a006f94e7 2024-03-26T02:00:13.139 INFO:teuthology.orchestra.run.smithi019.stdout:osd.0 smithi019 running (11m) 10s ago 11m 390M 1937M 17.2.0 e1d6a67b021e c1a797230ab6 2024-03-26T02:00:13.139 INFO:teuthology.orchestra.run.smithi019.stdout:osd.1 smithi019 running (11m) 10s ago 11m 578M 1937M 17.2.0 e1d6a67b021e 31d3489c7188 2024-03-26T02:00:13.139 INFO:teuthology.orchestra.run.smithi019.stdout:osd.2 smithi019 running (10m) 10s ago 10m 458M 1937M 17.2.0 e1d6a67b021e f549115e7685 2024-03-26T02:00:13.139 INFO:teuthology.orchestra.run.smithi019.stdout:osd.3 smithi019 running (10m) 10s ago 10m 584M 1937M 17.2.0 e1d6a67b021e 154af68d03e1 2024-03-26T02:00:13.139 INFO:teuthology.orchestra.run.smithi019.stdout:osd.4 smithi069 running (10m) 15s ago 10m 556M 2704M 17.2.0 e1d6a67b021e 4fd5e853f8bf 2024-03-26T02:00:13.139 INFO:teuthology.orchestra.run.smithi019.stdout:osd.5 smithi069 running (9m) 15s ago 9m 433M 2704M 17.2.0 e1d6a67b021e 932e401d87c9 2024-03-26T02:00:13.139 INFO:teuthology.orchestra.run.smithi019.stdout:osd.6 smithi069 running (9m) 15s ago 9m 345M 2704M 17.2.0 e1d6a67b021e b123f75154c8 2024-03-26T02:00:13.139 INFO:teuthology.orchestra.run.smithi019.stdout:osd.7 smithi069 running (8m) 15s ago 8m 550M 2704M 17.2.0 e1d6a67b021e d175fbc9ff8a 2024-03-26T02:00:13.139 INFO:teuthology.orchestra.run.smithi019.stdout:prometheus.smithi019 smithi019 *:9095 running (2m) 10s ago 12m 58.8M - 2.33.4 514e6a882f6e 29f89db72a07 2024-03-26T02:00:13.156 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:12 smithi019 ceph-mon[66359]: Upgrade: Updating osd.0 2024-03-26T02:00:13.156 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:12 smithi019 ceph-mon[66359]: Deploying daemon osd.0 on smithi019 2024-03-26T02:00:13.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:12 smithi069 ceph-mon[53087]: Upgrade: Updating osd.0 2024-03-26T02:00:13.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:12 smithi069 ceph-mon[53087]: Deploying daemon osd.0 on smithi019 2024-03-26T02:00:13.859 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:00:13.859 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": { 2024-03-26T02:00:13.859 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:00:13.859 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:00:13.859 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": { 2024-03-26T02:00:13.859 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:00:13.860 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:00:13.860 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": { 2024-03-26T02:00:13.860 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T02:00:13.860 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:00:13.860 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": { 2024-03-26T02:00:13.860 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:00:13.860 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:00:13.860 INFO:teuthology.orchestra.run.smithi019.stdout: "overall": { 2024-03-26T02:00:13.860 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-03-26T02:00:13.860 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 4 2024-03-26T02:00:13.860 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-26T02:00:13.860 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:00:14.110 INFO:teuthology.orchestra.run.smithi019.stdout: 5 198901 11.62 MB/sec execute 287 sec latency 35.144 ms 2024-03-26T02:00:14.167 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:13 smithi019 ceph-mon[66359]: from='client.44141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:00:14.167 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:13 smithi019 ceph-mon[66359]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:00:14.167 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:13 smithi019 ceph-mon[66359]: pgmap v54: 97 pgs: 97 active+clean; 339 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.5 MiB/s wr, 643 op/s 2024-03-26T02:00:14.168 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:13 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/2242807995' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:00:14.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:13 smithi069 ceph-mon[53087]: from='client.44141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:00:14.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:13 smithi069 ceph-mon[53087]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:00:14.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:13 smithi069 ceph-mon[53087]: pgmap v54: 97 pgs: 97 active+clean; 339 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.5 MiB/s wr, 643 op/s 2024-03-26T02:00:14.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:13 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/2242807995' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:00:14.397 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:00:14.397 INFO:teuthology.orchestra.run.smithi019.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2", 2024-03-26T02:00:14.397 INFO:teuthology.orchestra.run.smithi019.stdout: "in_progress": true, 2024-03-26T02:00:14.397 INFO:teuthology.orchestra.run.smithi019.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T02:00:14.397 INFO:teuthology.orchestra.run.smithi019.stdout: "services_complete": [ 2024-03-26T02:00:14.397 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr", 2024-03-26T02:00:14.398 INFO:teuthology.orchestra.run.smithi019.stdout: "crash", 2024-03-26T02:00:14.398 INFO:teuthology.orchestra.run.smithi019.stdout: "mon" 2024-03-26T02:00:14.398 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-26T02:00:14.398 INFO:teuthology.orchestra.run.smithi019.stdout: "progress": "6/23 daemons upgraded", 2024-03-26T02:00:14.398 INFO:teuthology.orchestra.run.smithi019.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T02:00:14.398 INFO:teuthology.orchestra.run.smithi019.stdout: "is_paused": false 2024-03-26T02:00:14.398 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:00:15.054 INFO:teuthology.orchestra.run.smithi019.stdout:HEALTH_OK 2024-03-26T02:00:15.110 INFO:teuthology.orchestra.run.smithi019.stdout: 5 199232 11.61 MB/sec execute 288 sec latency 45.235 ms 2024-03-26T02:00:15.178 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:15 smithi019 ceph-mon[66359]: from='client.44145 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:00:15.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:15 smithi069 ceph-mon[53087]: from='client.44145 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:00:16.110 INFO:teuthology.orchestra.run.smithi019.stdout: 5 199559 11.62 MB/sec execute 289 sec latency 72.148 ms 2024-03-26T02:00:16.191 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:16 smithi019 ceph-mon[66359]: from='client.34204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:00:16.191 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:16 smithi019 ceph-mon[66359]: pgmap v55: 97 pgs: 97 active+clean; 339 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.4 MiB/s wr, 436 op/s 2024-03-26T02:00:16.191 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:16 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/1101899750' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:00:16.192 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:16 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:00:16.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:16 smithi069 ceph-mon[53087]: from='client.34204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:00:16.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:16 smithi069 ceph-mon[53087]: pgmap v55: 97 pgs: 97 active+clean; 339 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.4 MiB/s wr, 436 op/s 2024-03-26T02:00:16.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:16 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/1101899750' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:00:16.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:16 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:00:17.110 INFO:teuthology.orchestra.run.smithi019.stdout: 5 199872 11.59 MB/sec execute 290 sec latency 71.665 ms 2024-03-26T02:00:17.481 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:17 smithi019 systemd[1]: Stopping Ceph osd.0 for 692607c6-eb12-11ee-95d0-87774f69a715... 2024-03-26T02:00:17.997 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:17 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0[35274]: 2024-03-26T02:00:17.862+0000 7f94f4966700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:00:17.998 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:17 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0[35274]: 2024-03-26T02:00:17.862+0000 7f94f4966700 -1 osd.0 73 *** Got signal Terminated *** 2024-03-26T02:00:17.998 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:17 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0[35274]: 2024-03-26T02:00:17.862+0000 7f94f4966700 -1 osd.0 73 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:00:18.110 INFO:teuthology.orchestra.run.smithi019.stdout: 5 200270 11.57 MB/sec execute 291 sec latency 105.254 ms 2024-03-26T02:00:18.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:18 smithi019 ceph-mon[66359]: pgmap v56: 97 pgs: 97 active+clean; 336 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 5.6 MiB/s wr, 509 op/s 2024-03-26T02:00:18.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:18 smithi019 ceph-mon[66359]: osd.0 marked itself down and dead 2024-03-26T02:00:18.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:18 smithi069 ceph-mon[53087]: pgmap v56: 97 pgs: 97 active+clean; 336 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 5.6 MiB/s wr, 509 op/s 2024-03-26T02:00:18.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:18 smithi069 ceph-mon[53087]: osd.0 marked itself down and dead 2024-03-26T02:00:18.990 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:18 smithi019 podman[72031]: 2024-03-26 02:00:18.684668088 +0000 UTC m=+1.008176983 container died c1a797230ab60e38182a9b8939aab844427cb3d77edfc36c8a56a61ead3ca861 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0, RELEASE=HEAD, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, distribution-scope=public, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, name=centos-stream, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, ceph=True, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc.) 2024-03-26T02:00:19.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:19 smithi069 ceph-mon[53087]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:00:19.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:19 smithi069 ceph-mon[53087]: failed to encode map e74 with expected crc 2024-03-26T02:00:19.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:19 smithi069 ceph-mon[53087]: osdmap e74: 8 total, 7 up, 8 in 2024-03-26T02:00:19.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:19 smithi069 ceph-mon[53087]: failed to encode map e74 with expected crc 2024-03-26T02:00:19.110 INFO:teuthology.orchestra.run.smithi019.stdout: 5 200285 11.53 MB/sec execute 292 sec latency 1079.413 ms 2024-03-26T02:00:19.246 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:19 smithi019 ceph-mon[66359]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:00:19.247 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:19 smithi019 ceph-mon[66359]: failed to encode map e74 with expected crc 2024-03-26T02:00:19.247 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:19 smithi019 ceph-mon[66359]: osdmap e74: 8 total, 7 up, 8 in 2024-03-26T02:00:19.247 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:19 smithi019 ceph-mon[66359]: failed to encode map e74 with expected crc 2024-03-26T02:00:19.759 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:19 smithi019 podman[72031]: 2024-03-26 02:00:19.507221114 +0000 UTC m=+1.830730006 container cleanup c1a797230ab60e38182a9b8939aab844427cb3d77edfc36c8a56a61ead3ca861 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, release=754, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, architecture=x86_64, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream) 2024-03-26T02:00:19.759 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:19 smithi019 bash[72031]: ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0 2024-03-26T02:00:20.010 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:19 smithi019 podman[72047]: 2024-03-26 02:00:19.894053349 +0000 UTC m=+1.208639716 container remove c1a797230ab60e38182a9b8939aab844427cb3d77edfc36c8a56a61ead3ca861 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0, io.openshift.tags=base centos centos-stream, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, vendor=Red Hat, Inc., architecture=x86_64, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, io.openshift.expose-services=, maintainer=Guillaume Abrioux , distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-26T02:00:20.111 INFO:teuthology.orchestra.run.smithi019.stdout: 5 200562 11.51 MB/sec execute 293 sec latency 35.344 ms 2024-03-26T02:00:20.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:20 smithi019 ceph-mon[66359]: pgmap v58: 97 pgs: 13 stale+active+clean, 84 active+clean; 340 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.3 MiB/s wr, 531 op/s 2024-03-26T02:00:20.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:20 smithi019 ceph-mon[66359]: failed to encode map e75 with expected crc 2024-03-26T02:00:20.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:20 smithi019 ceph-mon[66359]: osdmap e75: 8 total, 7 up, 8 in 2024-03-26T02:00:20.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:20 smithi019 ceph-mon[66359]: failed to encode map e75 with expected crc 2024-03-26T02:00:20.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:20 smithi019 ceph-mon[66359]: failed to encode map e75 with expected crc 2024-03-26T02:00:20.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:20 smithi019 ceph-mon[66359]: failed to encode map e75 with expected crc 2024-03-26T02:00:20.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:20 smithi019 ceph-mon[66359]: failed to encode map e75 with expected crc 2024-03-26T02:00:20.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:20 smithi019 ceph-mon[66359]: failed to encode map e75 with expected crc 2024-03-26T02:00:20.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:20 smithi019 ceph-mon[66359]: failed to encode map e75 with expected crc 2024-03-26T02:00:20.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:20 smithi019 ceph-mon[66359]: failed to encode map e75 with expected crc 2024-03-26T02:00:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:20 smithi069 ceph-mon[53087]: pgmap v58: 97 pgs: 13 stale+active+clean, 84 active+clean; 340 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.3 MiB/s wr, 531 op/s 2024-03-26T02:00:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:20 smithi069 ceph-mon[53087]: failed to encode map e75 with expected crc 2024-03-26T02:00:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:20 smithi069 ceph-mon[53087]: osdmap e75: 8 total, 7 up, 8 in 2024-03-26T02:00:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:20 smithi069 ceph-mon[53087]: failed to encode map e75 with expected crc 2024-03-26T02:00:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:20 smithi069 ceph-mon[53087]: failed to encode map e75 with expected crc 2024-03-26T02:00:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:20 smithi069 ceph-mon[53087]: failed to encode map e75 with expected crc 2024-03-26T02:00:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:20 smithi069 ceph-mon[53087]: failed to encode map e75 with expected crc 2024-03-26T02:00:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:20 smithi069 ceph-mon[53087]: failed to encode map e75 with expected crc 2024-03-26T02:00:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:20 smithi069 ceph-mon[53087]: failed to encode map e75 with expected crc 2024-03-26T02:00:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:20 smithi069 ceph-mon[53087]: failed to encode map e75 with expected crc 2024-03-26T02:00:20.518 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:20 smithi019 podman[72130]: 2024-03-26 02:00:20.319519775 +0000 UTC m=+0.026596137 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:00:20.519 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:20 smithi019 podman[72130]: 2024-03-26 02:00:20.504157124 +0000 UTC m=+0.211233509 container create 3f40397f5543dd6adfd67d8cf6fe4bf1531da2359709ff49270eb91889cc5e5d (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:00:21.110 INFO:teuthology.orchestra.run.smithi019.stdout: 5 201018 11.53 MB/sec execute 294 sec latency 34.849 ms 2024-03-26T02:00:21.284 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:21 smithi019 podman[72130]: 2024-03-26 02:00:21.111079178 +0000 UTC m=+0.818155521 container init 3f40397f5543dd6adfd67d8cf6fe4bf1531da2359709ff49270eb91889cc5e5d (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-26T02:00:21.285 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:21 smithi019 podman[72130]: 2024-03-26 02:00:21.117167682 +0000 UTC m=+0.824244028 container start 3f40397f5543dd6adfd67d8cf6fe4bf1531da2359709ff49270eb91889cc5e5d (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True) 2024-03-26T02:00:21.543 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:21 smithi019 podman[72130]: 2024-03-26 02:00:21.314535522 +0000 UTC m=+1.021611889 container attach 3f40397f5543dd6adfd67d8cf6fe4bf1531da2359709ff49270eb91889cc5e5d (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:00:21.793 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:21 smithi019 conmon[72144]: conmon 3f40397f5543dd6adfd6 : Failed to open cgroups file: /sys/fs/cgroup/machine.slice/libpod-3f40397f5543dd6adfd67d8cf6fe4bf1531da2359709ff49270eb91889cc5e5d.scope/container/memory.events 2024-03-26T02:00:21.794 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:21 smithi019 podman[72130]: 2024-03-26 02:00:21.557902228 +0000 UTC m=+1.264978571 container died 3f40397f5543dd6adfd67d8cf6fe4bf1531da2359709ff49270eb91889cc5e5d (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True) 2024-03-26T02:00:22.044 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:22 smithi019 ceph-mon[66359]: pgmap v60: 97 pgs: 13 stale+active+clean, 84 active+clean; 340 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.2 MiB/s wr, 309 op/s 2024-03-26T02:00:22.110 INFO:teuthology.orchestra.run.smithi019.stdout: 5 201538 11.56 MB/sec execute 295 sec latency 41.257 ms 2024-03-26T02:00:22.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:22 smithi069 ceph-mon[53087]: pgmap v60: 97 pgs: 13 stale+active+clean, 84 active+clean; 340 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.2 MiB/s wr, 309 op/s 2024-03-26T02:00:22.819 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:22 smithi019 podman[72155]: 2024-03-26 02:00:22.603158785 +0000 UTC m=+1.037228000 container remove 3f40397f5543dd6adfd67d8cf6fe4bf1531da2359709ff49270eb91889cc5e5d (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-26T02:00:22.819 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:22 smithi019 systemd[1]: ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.0.service: Deactivated successfully. 2024-03-26T02:00:22.819 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:22 smithi019 systemd[1]: Stopped Ceph osd.0 for 692607c6-eb12-11ee-95d0-87774f69a715. 2024-03-26T02:00:22.819 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:22 smithi019 systemd[1]: ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.0.service: Consumed 3min 38.425s CPU time. 2024-03-26T02:00:23.070 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:23 smithi019 ceph-mon[66359]: pgmap v61: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 346 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 6.1 MiB/s wr, 589 op/s; 229/1707 objects degraded (13.415%) 2024-03-26T02:00:23.071 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:22 smithi019 systemd[1]: Starting Ceph osd.0 for 692607c6-eb12-11ee-95d0-87774f69a715... 2024-03-26T02:00:23.110 INFO:teuthology.orchestra.run.smithi019.stdout: 5 201878 11.54 MB/sec execute 296 sec latency 40.482 ms 2024-03-26T02:00:23.322 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:23 smithi019 podman[72256]: 2024-03-26 02:00:23.105559903 +0000 UTC m=+0.022241351 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:00:23.322 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:23 smithi019 podman[72256]: 2024-03-26 02:00:23.305490959 +0000 UTC m=+0.222172395 container create fb3121be1eda69536ff1147d885eeffdaf99c79028dd3011fc9f146c63724366 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0-activate, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-26T02:00:23.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:23 smithi069 ceph-mon[53087]: pgmap v61: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 346 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 6.1 MiB/s wr, 589 op/s; 229/1707 objects degraded (13.415%) 2024-03-26T02:00:23.949 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:23 smithi019 podman[72256]: 2024-03-26 02:00:23.706536955 +0000 UTC m=+0.623218399 container init fb3121be1eda69536ff1147d885eeffdaf99c79028dd3011fc9f146c63724366 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-26T02:00:23.950 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:23 smithi019 podman[72256]: 2024-03-26 02:00:23.712482246 +0000 UTC m=+0.629163686 container start fb3121be1eda69536ff1147d885eeffdaf99c79028dd3011fc9f146c63724366 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0-activate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:00:23.950 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:23 smithi019 podman[72256]: 2024-03-26 02:00:23.818388073 +0000 UTC m=+0.735069529 container attach fb3121be1eda69536ff1147d885eeffdaf99c79028dd3011fc9f146c63724366 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0-activate, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-26T02:00:24.110 INFO:teuthology.orchestra.run.smithi019.stdout: 5 202095 11.53 MB/sec execute 297 sec latency 69.117 ms 2024-03-26T02:00:24.200 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:24 smithi019 ceph-mon[66359]: Health check failed: Degraded data redundancy: 229/1707 objects degraded (13.415%), 29 pgs degraded (PG_DEGRADED) 2024-03-26T02:00:24.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:24 smithi069 ceph-mon[53087]: Health check failed: Degraded data redundancy: 229/1707 objects degraded (13.415%), 29 pgs degraded (PG_DEGRADED) 2024-03-26T02:00:24.452 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:24 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0-activate[72270]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-26T02:00:24.452 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:24 smithi019 bash[72256]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-26T02:00:24.452 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:24 smithi019 bash[72256]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-26T02:00:24.452 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:24 smithi019 bash[72256]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-26T02:00:24.452 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:24 smithi019 bash[72256]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-26T02:00:24.453 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:24 smithi019 bash[72256]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-26T02:00:24.453 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:24 smithi019 bash[72256]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-26T02:00:24.453 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:24 smithi019 bash[72256]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-26T02:00:24.453 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:24 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0-activate[72270]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-26T02:00:24.453 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:24 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0-activate[72270]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-26T02:00:24.453 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:24 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0-activate[72270]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-26T02:00:24.453 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:24 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0-activate[72270]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-26T02:00:24.453 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:24 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0-activate[72270]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-26T02:00:24.453 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:24 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0-activate[72270]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-26T02:00:24.453 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:24 smithi019 podman[72256]: 2024-03-26 02:00:24.244933704 +0000 UTC m=+1.161615148 container died fb3121be1eda69536ff1147d885eeffdaf99c79028dd3011fc9f146c63724366 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0-activate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-5e70124, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:00:25.110 INFO:teuthology.orchestra.run.smithi019.stdout: 5 202380 11.50 MB/sec execute 298 sec latency 73.322 ms 2024-03-26T02:00:25.253 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:25 smithi019 ceph-mon[66359]: pgmap v62: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 346 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 4.3 MiB/s wr, 478 op/s; 229/1707 objects degraded (13.415%) 2024-03-26T02:00:25.358 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:25 smithi069 ceph-mon[53087]: pgmap v62: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 346 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 4.3 MiB/s wr, 478 op/s; 229/1707 objects degraded (13.415%) 2024-03-26T02:00:25.513 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:25 smithi019 podman[72407]: 2024-03-26 02:00:25.277191007 +0000 UTC m=+1.026875003 container remove fb3121be1eda69536ff1147d885eeffdaf99c79028dd3011fc9f146c63724366 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0-activate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-26T02:00:25.765 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:25 smithi019 podman[72496]: 2024-03-26 02:00:25.418226924 +0000 UTC m=+0.022213575 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:00:25.765 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:25 smithi019 podman[72496]: 2024-03-26 02:00:25.555879616 +0000 UTC m=+0.159866254 container create 3f931cfe320ac0caf9266b9378644dbdc715b0d5494045f3906ee69686a30a07 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-26T02:00:26.018 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:25 smithi019 podman[72496]: 2024-03-26 02:00:25.982804776 +0000 UTC m=+0.586791417 container init 3f931cfe320ac0caf9266b9378644dbdc715b0d5494045f3906ee69686a30a07 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-26T02:00:26.019 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:25 smithi019 podman[72496]: 2024-03-26 02:00:25.986986918 +0000 UTC m=+0.590973564 container start 3f931cfe320ac0caf9266b9378644dbdc715b0d5494045f3906ee69686a30a07 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-26T02:00:26.019 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:26 smithi019 ceph-osd[72515]: -- 172.21.15.19:0/2146712328 <== mon.0 v2:172.21.15.19:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55df4e9971e0 con 0x55df4e969000 2024-03-26T02:00:26.110 INFO:teuthology.orchestra.run.smithi019.stdout: 5 202640 11.49 MB/sec execute 299 sec latency 54.636 ms 2024-03-26T02:00:26.274 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:26 smithi019 bash[72496]: 3f931cfe320ac0caf9266b9378644dbdc715b0d5494045f3906ee69686a30a07 2024-03-26T02:00:26.274 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:26 smithi019 systemd[1]: Started Ceph osd.0 for 692607c6-eb12-11ee-95d0-87774f69a715. 2024-03-26T02:00:26.816 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:26 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0[72511]: 2024-03-26T02:00:26.559+0000 7fb56da717c0 -1 Falling back to public interface 2024-03-26T02:00:27.110 INFO:teuthology.orchestra.run.smithi019.stdout: 5 202960 11.49 MB/sec execute 300 sec latency 53.575 ms 2024-03-26T02:00:27.332 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:27 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:27.332 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:27 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:27.333 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:27 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:00:27.333 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:27 smithi019 ceph-mon[66359]: pgmap v63: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 347 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 2.8 KiB/s rd, 5.3 MiB/s wr, 506 op/s; 237/1737 objects degraded (13.644%) 2024-03-26T02:00:27.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:27 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:27.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:27 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:27.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:27 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:00:27.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:27 smithi069 ceph-mon[53087]: pgmap v63: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 347 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 2.8 KiB/s rd, 5.3 MiB/s wr, 506 op/s; 237/1737 objects degraded (13.644%) 2024-03-26T02:00:28.111 INFO:teuthology.orchestra.run.smithi019.stdout: 5 203527 11.49 MB/sec execute 301 sec latency 84.596 ms 2024-03-26T02:00:29.111 INFO:teuthology.orchestra.run.smithi019.stdout: 5 203994 11.47 MB/sec execute 302 sec latency 27.785 ms 2024-03-26T02:00:29.860 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:29 smithi019 ceph-mon[66359]: pgmap v64: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 347 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.7 MiB/s wr, 449 op/s; 237/1737 objects degraded (13.644%) 2024-03-26T02:00:29.860 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:29 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 237/1737 objects degraded (13.644%), 29 pgs degraded (PG_DEGRADED) 2024-03-26T02:00:30.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:29 smithi069 ceph-mon[53087]: pgmap v64: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 347 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.7 MiB/s wr, 449 op/s; 237/1737 objects degraded (13.644%) 2024-03-26T02:00:30.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:29 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 237/1737 objects degraded (13.644%), 29 pgs degraded (PG_DEGRADED) 2024-03-26T02:00:30.110 INFO:teuthology.orchestra.run.smithi019.stdout: 5 204509 11.50 MB/sec execute 303 sec latency 48.920 ms 2024-03-26T02:00:31.111 INFO:teuthology.orchestra.run.smithi019.stdout: 5 205083 11.52 MB/sec execute 304 sec latency 31.190 ms 2024-03-26T02:00:31.876 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:31 smithi019 ceph-mon[66359]: pgmap v65: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 347 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.0 MiB/s wr, 385 op/s; 237/1737 objects degraded (13.644%) 2024-03-26T02:00:31.877 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:31 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:31.877 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:31 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:00:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:31 smithi069 ceph-mon[53087]: pgmap v65: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 347 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.0 MiB/s wr, 385 op/s; 237/1737 objects degraded (13.644%) 2024-03-26T02:00:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:31 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:31 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:00:32.110 INFO:teuthology.orchestra.run.smithi019.stdout: 5 205451 11.52 MB/sec execute 305 sec latency 39.507 ms 2024-03-26T02:00:33.111 INFO:teuthology.orchestra.run.smithi019.stdout: 5 205740 11.50 MB/sec execute 306 sec latency 56.210 ms 2024-03-26T02:00:33.921 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:33 smithi019 ceph-mon[66359]: pgmap v66: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 345 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 8.8 KiB/s rd, 6.8 MiB/s wr, 674 op/s; 232/1728 objects degraded (13.426%) 2024-03-26T02:00:34.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:33 smithi069 ceph-mon[53087]: pgmap v66: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 345 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 8.8 KiB/s rd, 6.8 MiB/s wr, 674 op/s; 232/1728 objects degraded (13.426%) 2024-03-26T02:00:34.111 INFO:teuthology.orchestra.run.smithi019.stdout: 5 206078 11.50 MB/sec execute 307 sec latency 49.398 ms 2024-03-26T02:00:34.437 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:34 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0[72511]: 2024-03-26T02:00:34.397+0000 7fb56da717c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-03-26T02:00:34.942 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:34 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 232/1728 objects degraded (13.426%), 29 pgs degraded (PG_DEGRADED) 2024-03-26T02:00:35.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:34 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 232/1728 objects degraded (13.426%), 29 pgs degraded (PG_DEGRADED) 2024-03-26T02:00:35.111 INFO:teuthology.orchestra.run.smithi019.stdout: 5 206395 11.49 MB/sec execute 308 sec latency 44.480 ms 2024-03-26T02:00:35.955 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:35 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0[72511]: 2024-03-26T02:00:35.944+0000 7fb56da717c0 -1 osd.0 73 log_to_monitors true 2024-03-26T02:00:35.955 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:35 smithi019 ceph-mon[66359]: pgmap v67: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 345 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.9 MiB/s wr, 487 op/s; 232/1728 objects degraded (13.426%) 2024-03-26T02:00:35.955 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:35 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:35.955 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:35 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:35 smithi069 ceph-mon[53087]: pgmap v67: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 345 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.9 MiB/s wr, 487 op/s; 232/1728 objects degraded (13.426%) 2024-03-26T02:00:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:35 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:35 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:36.111 INFO:teuthology.orchestra.run.smithi019.stdout: 5 206767 11.50 MB/sec execute 309 sec latency 34.641 ms 2024-03-26T02:00:36.985 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:00:36 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0[72511]: 2024-03-26T02:00:36.829+0000 7fb55eec9700 -1 osd.0 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:00:36.985 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:36 smithi019 ceph-mon[66359]: from='osd.0 [v2:172.21.15.19:6802/2081454105,v1:172.21.15.19:6803/2081454105]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T02:00:37.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:36 smithi069 ceph-mon[53087]: from='osd.0 [v2:172.21.15.19:6802/2081454105,v1:172.21.15.19:6803/2081454105]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T02:00:37.111 INFO:teuthology.orchestra.run.smithi019.stdout: 5 207158 11.47 MB/sec execute 310 sec latency 73.864 ms 2024-03-26T02:00:38.000 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:37 smithi019 ceph-mon[66359]: pgmap v68: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 342 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 6.7 MiB/s wr, 685 op/s; 214/1623 objects degraded (13.185%) 2024-03-26T02:00:38.000 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:37 smithi019 ceph-mon[66359]: from='osd.0 [v2:172.21.15.19:6802/2081454105,v1:172.21.15.19:6803/2081454105]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T02:00:38.000 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:37 smithi019 ceph-mon[66359]: failed to encode map e76 with expected crc 2024-03-26T02:00:38.000 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:37 smithi019 ceph-mon[66359]: osdmap e76: 8 total, 7 up, 8 in 2024-03-26T02:00:38.000 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:37 smithi019 ceph-mon[66359]: from='osd.0 [v2:172.21.15.19:6802/2081454105,v1:172.21.15.19:6803/2081454105]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-03-26T02:00:38.000 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:37 smithi019 ceph-mon[66359]: failed to encode map e76 with expected crc 2024-03-26T02:00:38.000 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:37 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:38.000 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:37 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:38.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:37 smithi069 ceph-mon[53087]: pgmap v68: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 342 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 6.7 MiB/s wr, 685 op/s; 214/1623 objects degraded (13.185%) 2024-03-26T02:00:38.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:37 smithi069 ceph-mon[53087]: from='osd.0 [v2:172.21.15.19:6802/2081454105,v1:172.21.15.19:6803/2081454105]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T02:00:38.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:37 smithi069 ceph-mon[53087]: failed to encode map e76 with expected crc 2024-03-26T02:00:38.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:37 smithi069 ceph-mon[53087]: osdmap e76: 8 total, 7 up, 8 in 2024-03-26T02:00:38.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:37 smithi069 ceph-mon[53087]: from='osd.0 [v2:172.21.15.19:6802/2081454105,v1:172.21.15.19:6803/2081454105]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-03-26T02:00:38.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:37 smithi069 ceph-mon[53087]: failed to encode map e76 with expected crc 2024-03-26T02:00:38.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:37 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:38.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:37 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:38.111 INFO:teuthology.orchestra.run.smithi019.stdout: 5 207499 11.45 MB/sec execute 311 sec latency 315.561 ms 2024-03-26T02:00:39.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:38 smithi019 ceph-mon[66359]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:00:39.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:38 smithi019 ceph-mon[66359]: failed to encode map e77 with expected crc 2024-03-26T02:00:39.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:38 smithi019 ceph-mon[66359]: osd.0 [v2:172.21.15.19:6802/2081454105,v1:172.21.15.19:6803/2081454105] boot 2024-03-26T02:00:39.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:38 smithi019 ceph-mon[66359]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T02:00:39.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:38 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:00:39.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:38 smithi019 ceph-mon[66359]: failed to encode map e77 with expected crc 2024-03-26T02:00:39.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:38 smithi069 ceph-mon[53087]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:00:39.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:38 smithi069 ceph-mon[53087]: failed to encode map e77 with expected crc 2024-03-26T02:00:39.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:38 smithi069 ceph-mon[53087]: osd.0 [v2:172.21.15.19:6802/2081454105,v1:172.21.15.19:6803/2081454105] boot 2024-03-26T02:00:39.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:38 smithi069 ceph-mon[53087]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T02:00:39.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:38 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:00:39.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:38 smithi069 ceph-mon[53087]: failed to encode map e77 with expected crc 2024-03-26T02:00:39.111 INFO:teuthology.orchestra.run.smithi019.stdout: 5 207499 11.41 MB/sec execute 312 sec latency 1315.606 ms 2024-03-26T02:00:39.986 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:39 smithi019 ceph-mon[66359]: pgmap v71: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 338 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 765 op/s; 211/1620 objects degraded (13.025%) 2024-03-26T02:00:39.987 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:39 smithi019 ceph-mon[66359]: failed to encode map e78 with expected crc 2024-03-26T02:00:39.987 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:39 smithi019 ceph-mon[66359]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T02:00:39.987 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:39 smithi019 ceph-mon[66359]: failed to encode map e78 with expected crc 2024-03-26T02:00:39.987 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:39 smithi019 ceph-mon[66359]: failed to encode map e78 with expected crc 2024-03-26T02:00:39.987 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:39 smithi019 ceph-mon[66359]: failed to encode map e78 with expected crc 2024-03-26T02:00:39.987 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:39 smithi019 ceph-mon[66359]: failed to encode map e78 with expected crc 2024-03-26T02:00:39.987 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:39 smithi019 ceph-mon[66359]: failed to encode map e78 with expected crc 2024-03-26T02:00:39.987 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:39 smithi019 ceph-mon[66359]: failed to encode map e78 with expected crc 2024-03-26T02:00:39.987 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:39 smithi019 ceph-mon[66359]: failed to encode map e78 with expected crc 2024-03-26T02:00:39.987 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:39 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 211/1620 objects degraded (13.025%), 29 pgs degraded (PG_DEGRADED) 2024-03-26T02:00:40.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:39 smithi069 ceph-mon[53087]: pgmap v71: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 338 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 765 op/s; 211/1620 objects degraded (13.025%) 2024-03-26T02:00:40.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:39 smithi069 ceph-mon[53087]: failed to encode map e78 with expected crc 2024-03-26T02:00:40.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:39 smithi069 ceph-mon[53087]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T02:00:40.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:39 smithi069 ceph-mon[53087]: failed to encode map e78 with expected crc 2024-03-26T02:00:40.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:39 smithi069 ceph-mon[53087]: failed to encode map e78 with expected crc 2024-03-26T02:00:40.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:39 smithi069 ceph-mon[53087]: failed to encode map e78 with expected crc 2024-03-26T02:00:40.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:39 smithi069 ceph-mon[53087]: failed to encode map e78 with expected crc 2024-03-26T02:00:40.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:39 smithi069 ceph-mon[53087]: failed to encode map e78 with expected crc 2024-03-26T02:00:40.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:39 smithi069 ceph-mon[53087]: failed to encode map e78 with expected crc 2024-03-26T02:00:40.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:39 smithi069 ceph-mon[53087]: failed to encode map e78 with expected crc 2024-03-26T02:00:40.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:39 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 211/1620 objects degraded (13.025%), 29 pgs degraded (PG_DEGRADED) 2024-03-26T02:00:40.111 INFO:teuthology.orchestra.run.smithi019.stdout: 5 207506 11.37 MB/sec execute 313 sec latency 2298.347 ms 2024-03-26T02:00:41.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:40 smithi019 ceph-mon[66359]: failed to encode map e79 with expected crc 2024-03-26T02:00:41.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:40 smithi019 ceph-mon[66359]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T02:00:41.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:40 smithi019 ceph-mon[66359]: failed to encode map e79 with expected crc 2024-03-26T02:00:41.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:40 smithi019 ceph-mon[66359]: failed to encode map e79 with expected crc 2024-03-26T02:00:41.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:40 smithi019 ceph-mon[66359]: failed to encode map e79 with expected crc 2024-03-26T02:00:41.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:40 smithi019 ceph-mon[66359]: failed to encode map e79 with expected crc 2024-03-26T02:00:41.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:40 smithi019 ceph-mon[66359]: failed to encode map e79 with expected crc 2024-03-26T02:00:41.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:40 smithi019 ceph-mon[66359]: failed to encode map e79 with expected crc 2024-03-26T02:00:41.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:40 smithi069 ceph-mon[53087]: failed to encode map e79 with expected crc 2024-03-26T02:00:41.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:40 smithi069 ceph-mon[53087]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T02:00:41.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:40 smithi069 ceph-mon[53087]: failed to encode map e79 with expected crc 2024-03-26T02:00:41.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:40 smithi069 ceph-mon[53087]: failed to encode map e79 with expected crc 2024-03-26T02:00:41.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:40 smithi069 ceph-mon[53087]: failed to encode map e79 with expected crc 2024-03-26T02:00:41.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:40 smithi069 ceph-mon[53087]: failed to encode map e79 with expected crc 2024-03-26T02:00:41.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:40 smithi069 ceph-mon[53087]: failed to encode map e79 with expected crc 2024-03-26T02:00:41.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:40 smithi069 ceph-mon[53087]: failed to encode map e79 with expected crc 2024-03-26T02:00:41.111 INFO:teuthology.orchestra.run.smithi019.stdout: 5 207695 11.35 MB/sec execute 314 sec latency 266.382 ms 2024-03-26T02:00:42.011 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:41 smithi019 ceph-mon[66359]: pgmap v74: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 338 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 590 KiB/s wr, 36 op/s; 211/1620 objects degraded (13.025%) 2024-03-26T02:00:42.011 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:41 smithi019 ceph-mon[66359]: failed to encode map e80 with expected crc 2024-03-26T02:00:42.011 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:41 smithi019 ceph-mon[66359]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T02:00:42.011 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:41 smithi019 ceph-mon[66359]: failed to encode map e80 with expected crc 2024-03-26T02:00:42.011 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:41 smithi019 ceph-mon[66359]: failed to encode map e80 with expected crc 2024-03-26T02:00:42.011 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:41 smithi019 ceph-mon[66359]: failed to encode map e80 with expected crc 2024-03-26T02:00:42.011 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:41 smithi019 ceph-mon[66359]: failed to encode map e80 with expected crc 2024-03-26T02:00:42.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:41 smithi069 ceph-mon[53087]: pgmap v74: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 338 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 590 KiB/s wr, 36 op/s; 211/1620 objects degraded (13.025%) 2024-03-26T02:00:42.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:41 smithi069 ceph-mon[53087]: failed to encode map e80 with expected crc 2024-03-26T02:00:42.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:41 smithi069 ceph-mon[53087]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T02:00:42.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:41 smithi069 ceph-mon[53087]: failed to encode map e80 with expected crc 2024-03-26T02:00:42.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:41 smithi069 ceph-mon[53087]: failed to encode map e80 with expected crc 2024-03-26T02:00:42.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:41 smithi069 ceph-mon[53087]: failed to encode map e80 with expected crc 2024-03-26T02:00:42.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:41 smithi069 ceph-mon[53087]: failed to encode map e80 with expected crc 2024-03-26T02:00:42.111 INFO:teuthology.orchestra.run.smithi019.stdout: 5 207772 11.33 MB/sec execute 315 sec latency 1053.780 ms 2024-03-26T02:00:43.022 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:42 smithi019 ceph-mon[66359]: failed to encode map e81 with expected crc 2024-03-26T02:00:43.022 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:42 smithi019 ceph-mon[66359]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T02:00:43.022 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:42 smithi019 ceph-mon[66359]: failed to encode map e81 with expected crc 2024-03-26T02:00:43.022 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:42 smithi019 ceph-mon[66359]: failed to encode map e81 with expected crc 2024-03-26T02:00:43.022 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:42 smithi019 ceph-mon[66359]: failed to encode map e81 with expected crc 2024-03-26T02:00:43.022 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:42 smithi019 ceph-mon[66359]: failed to encode map e81 with expected crc 2024-03-26T02:00:43.022 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:42 smithi019 ceph-mon[66359]: failed to encode map e81 with expected crc 2024-03-26T02:00:43.022 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:42 smithi019 ceph-mon[66359]: failed to encode map e81 with expected crc 2024-03-26T02:00:43.022 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:42 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:43.023 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:42 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:43.023 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:42 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:43.023 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:42 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:00:43.023 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:42 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:43.023 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:42 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:43.023 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:42 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:00:43.023 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:42 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:00:43.023 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:42 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:00:43.023 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:42 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:00:43.023 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:42 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T02:00:43.111 INFO:teuthology.orchestra.run.smithi019.stdout: 5 208163 11.34 MB/sec execute 316 sec latency 55.226 ms 2024-03-26T02:00:43.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:42 smithi069 ceph-mon[53087]: failed to encode map e81 with expected crc 2024-03-26T02:00:43.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:42 smithi069 ceph-mon[53087]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T02:00:43.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:42 smithi069 ceph-mon[53087]: failed to encode map e81 with expected crc 2024-03-26T02:00:43.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:42 smithi069 ceph-mon[53087]: failed to encode map e81 with expected crc 2024-03-26T02:00:43.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:42 smithi069 ceph-mon[53087]: failed to encode map e81 with expected crc 2024-03-26T02:00:43.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:42 smithi069 ceph-mon[53087]: failed to encode map e81 with expected crc 2024-03-26T02:00:43.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:42 smithi069 ceph-mon[53087]: failed to encode map e81 with expected crc 2024-03-26T02:00:43.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:42 smithi069 ceph-mon[53087]: failed to encode map e81 with expected crc 2024-03-26T02:00:43.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:42 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:43.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:42 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:43.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:42 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:43.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:42 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:00:43.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:42 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:43.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:42 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:43.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:42 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:00:43.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:42 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:00:43.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:42 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:00:43.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:42 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:00:43.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:42 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T02:00:44.038 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:43 smithi019 ceph-mon[66359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T02:00:44.039 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:43 smithi019 ceph-mon[66359]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-03-26T02:00:44.039 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:43 smithi019 ceph-mon[66359]: pgmap v77: 97 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 6 peering, 1 active+recovering+undersized+degraded+remapped, 83 active+clean; 329 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.0 KiB/s rd, 2.8 MiB/s wr, 317 op/s; 189/1443 objects degraded (13.098%); 1.6 MiB/s, 17 objects/s recovering 2024-03-26T02:00:44.039 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:43 smithi019 ceph-mon[66359]: failed to encode map e82 with expected crc 2024-03-26T02:00:44.039 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:43 smithi019 ceph-mon[66359]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T02:00:44.039 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:43 smithi019 ceph-mon[66359]: failed to encode map e82 with expected crc 2024-03-26T02:00:44.039 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:43 smithi019 ceph-mon[66359]: failed to encode map e82 with expected crc 2024-03-26T02:00:44.039 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:43 smithi019 ceph-mon[66359]: failed to encode map e82 with expected crc 2024-03-26T02:00:44.111 INFO:teuthology.orchestra.run.smithi019.stdout: 5 208552 11.34 MB/sec execute 317 sec latency 817.058 ms 2024-03-26T02:00:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:43 smithi069 ceph-mon[53087]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T02:00:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:43 smithi069 ceph-mon[53087]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-03-26T02:00:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:43 smithi069 ceph-mon[53087]: pgmap v77: 97 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 6 peering, 1 active+recovering+undersized+degraded+remapped, 83 active+clean; 329 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.0 KiB/s rd, 2.8 MiB/s wr, 317 op/s; 189/1443 objects degraded (13.098%); 1.6 MiB/s, 17 objects/s recovering 2024-03-26T02:00:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:43 smithi069 ceph-mon[53087]: failed to encode map e82 with expected crc 2024-03-26T02:00:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:43 smithi069 ceph-mon[53087]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T02:00:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:43 smithi069 ceph-mon[53087]: failed to encode map e82 with expected crc 2024-03-26T02:00:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:43 smithi069 ceph-mon[53087]: failed to encode map e82 with expected crc 2024-03-26T02:00:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:43 smithi069 ceph-mon[53087]: failed to encode map e82 with expected crc 2024-03-26T02:00:45.054 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:44 smithi019 ceph-mon[66359]: failed to encode map e83 with expected crc 2024-03-26T02:00:45.055 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:44 smithi019 ceph-mon[66359]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T02:00:45.055 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:44 smithi019 ceph-mon[66359]: failed to encode map e83 with expected crc 2024-03-26T02:00:45.055 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:44 smithi019 ceph-mon[66359]: failed to encode map e83 with expected crc 2024-03-26T02:00:45.055 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:44 smithi019 ceph-mon[66359]: failed to encode map e83 with expected crc 2024-03-26T02:00:45.055 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:44 smithi019 ceph-mon[66359]: failed to encode map e83 with expected crc 2024-03-26T02:00:45.055 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:44 smithi019 ceph-mon[66359]: failed to encode map e83 with expected crc 2024-03-26T02:00:45.055 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:44 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 189/1443 objects degraded (13.098%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T02:00:45.055 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:44 smithi019 ceph-mon[66359]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T02:00:45.111 INFO:teuthology.orchestra.run.smithi019.stdout: 5 208910 11.34 MB/sec execute 318 sec latency 73.018 ms 2024-03-26T02:00:45.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:44 smithi069 ceph-mon[53087]: failed to encode map e83 with expected crc 2024-03-26T02:00:45.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:44 smithi069 ceph-mon[53087]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T02:00:45.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:44 smithi069 ceph-mon[53087]: failed to encode map e83 with expected crc 2024-03-26T02:00:45.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:44 smithi069 ceph-mon[53087]: failed to encode map e83 with expected crc 2024-03-26T02:00:45.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:44 smithi069 ceph-mon[53087]: failed to encode map e83 with expected crc 2024-03-26T02:00:45.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:44 smithi069 ceph-mon[53087]: failed to encode map e83 with expected crc 2024-03-26T02:00:45.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:44 smithi069 ceph-mon[53087]: failed to encode map e83 with expected crc 2024-03-26T02:00:45.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:44 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 189/1443 objects degraded (13.098%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T02:00:45.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:44 smithi069 ceph-mon[53087]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T02:00:45.536 INFO:teuthology.orchestra.run.smithi019.stdout:true 2024-03-26T02:00:46.083 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:45 smithi019 ceph-mon[66359]: pgmap v80: 97 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 6 peering, 1 active+recovering+undersized+degraded+remapped, 83 active+clean; 329 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.0 KiB/s rd, 2.8 MiB/s wr, 317 op/s; 189/1443 objects degraded (13.098%); 1.6 MiB/s, 17 objects/s recovering 2024-03-26T02:00:46.084 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:45 smithi019 ceph-mon[66359]: failed to encode map e84 with expected crc 2024-03-26T02:00:46.084 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:45 smithi019 ceph-mon[66359]: failed to encode map e84 with expected crc 2024-03-26T02:00:46.084 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:45 smithi019 ceph-mon[66359]: failed to encode map e84 with expected crc 2024-03-26T02:00:46.084 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:45 smithi019 ceph-mon[66359]: failed to encode map e84 with expected crc 2024-03-26T02:00:46.084 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:45 smithi019 ceph-mon[66359]: failed to encode map e84 with expected crc 2024-03-26T02:00:46.084 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:45 smithi019 ceph-mon[66359]: failed to encode map e84 with expected crc 2024-03-26T02:00:46.084 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:46.084 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:00:46.084 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:45 smithi019 ceph-mon[66359]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T02:00:46.111 INFO:teuthology.orchestra.run.smithi019.stdout: 5 209073 11.32 MB/sec execute 319 sec latency 879.715 ms 2024-03-26T02:00:46.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:45 smithi069 ceph-mon[53087]: pgmap v80: 97 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 6 peering, 1 active+recovering+undersized+degraded+remapped, 83 active+clean; 329 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.0 KiB/s rd, 2.8 MiB/s wr, 317 op/s; 189/1443 objects degraded (13.098%); 1.6 MiB/s, 17 objects/s recovering 2024-03-26T02:00:46.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:45 smithi069 ceph-mon[53087]: failed to encode map e84 with expected crc 2024-03-26T02:00:46.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:45 smithi069 ceph-mon[53087]: failed to encode map e84 with expected crc 2024-03-26T02:00:46.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:45 smithi069 ceph-mon[53087]: failed to encode map e84 with expected crc 2024-03-26T02:00:46.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:45 smithi069 ceph-mon[53087]: failed to encode map e84 with expected crc 2024-03-26T02:00:46.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:45 smithi069 ceph-mon[53087]: failed to encode map e84 with expected crc 2024-03-26T02:00:46.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:45 smithi069 ceph-mon[53087]: failed to encode map e84 with expected crc 2024-03-26T02:00:46.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:00:46.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:00:46.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:45 smithi069 ceph-mon[53087]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T02:00:46.403 INFO:teuthology.orchestra.run.smithi019.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:00:46.403 INFO:teuthology.orchestra.run.smithi019.stdout:alertmanager.smithi019 smithi019 *:9093,9094 running (4m) 11s ago 13m 14.4M - 0.23.0 ba2b418f427c a6630841c2b2 2024-03-26T02:00:46.403 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi019 smithi019 running (59s) 11s ago 13m 6660k - 19.0.0-1775-g5e70124b 5f21c9264136 c5f91a47ddf2 2024-03-26T02:00:46.403 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi069 smithi069 running (53s) 48s ago 13m 6669k - 19.0.0-1775-g5e70124b 5f21c9264136 05128803b49e 2024-03-26T02:00:46.403 INFO:teuthology.orchestra.run.smithi019.stdout:grafana.smithi019 smithi019 *:3000 running (4m) 11s ago 13m 47.3M - 8.3.5 dad864ee21e9 a97dea3b5fec 2024-03-26T02:00:46.403 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi019.lmryom smithi019 running (7m) 11s ago 7m 38.1M - 17.2.0 e1d6a67b021e 91a7c2942715 2024-03-26T02:00:46.404 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi069.spgwgo smithi069 running (7m) 48s ago 7m 14.4M - 17.2.0 e1d6a67b021e 94d96338dbe1 2024-03-26T02:00:46.404 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi019.ixuvdm smithi019 *:8443,9283,8765 running (3m) 11s ago 15m 499M - 19.0.0-1775-g5e70124b 5f21c9264136 4460b22c7a56 2024-03-26T02:00:46.404 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi069.wpbewu smithi069 *:8443,9283,8765 running (2m) 48s ago 13m 451M - 19.0.0-1775-g5e70124b 5f21c9264136 380fd33842a8 2024-03-26T02:00:46.404 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi019 smithi019 running (2m) 11s ago 15m 47.6M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 c18934705774 2024-03-26T02:00:46.404 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi069 smithi069 running (116s) 48s ago 13m 35.6M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 a856bd263104 2024-03-26T02:00:46.404 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.0.0.smithi019.qbizxo smithi019 *:2049 running (7m) 11s ago 7m 284M - 4.0 e1d6a67b021e cd6628a9075d 2024-03-26T02:00:46.404 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.1.0.smithi069.uawdhc smithi069 *:2049 running (7m) 48s ago 7m 85.3M - 4.0 e1d6a67b021e 6ee3695e7a10 2024-03-26T02:00:46.404 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi019 smithi019 *:9100 running (4m) 11s ago 13m 10.2M - 1.3.1 1dbe0e931976 0d1d184bec63 2024-03-26T02:00:46.404 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi069 smithi069 *:9100 running (3m) 48s ago 12m 19.2M - 1.3.1 1dbe0e931976 c49a006f94e7 2024-03-26T02:00:46.404 INFO:teuthology.orchestra.run.smithi019.stdout:osd.0 smithi019 running (20s) 11s ago 12m 12.6M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 3f931cfe320a 2024-03-26T02:00:46.404 INFO:teuthology.orchestra.run.smithi019.stdout:osd.1 smithi019 running (11m) 11s ago 11m 348M 1937M 17.2.0 e1d6a67b021e 31d3489c7188 2024-03-26T02:00:46.404 INFO:teuthology.orchestra.run.smithi019.stdout:osd.2 smithi019 running (11m) 11s ago 11m 483M 1937M 17.2.0 e1d6a67b021e f549115e7685 2024-03-26T02:00:46.404 INFO:teuthology.orchestra.run.smithi019.stdout:osd.3 smithi019 running (10m) 11s ago 10m 601M 1937M 17.2.0 e1d6a67b021e 154af68d03e1 2024-03-26T02:00:46.404 INFO:teuthology.orchestra.run.smithi019.stdout:osd.4 smithi069 running (10m) 48s ago 10m 556M 2704M 17.2.0 e1d6a67b021e 4fd5e853f8bf 2024-03-26T02:00:46.404 INFO:teuthology.orchestra.run.smithi019.stdout:osd.5 smithi069 running (10m) 48s ago 10m 433M 2704M 17.2.0 e1d6a67b021e 932e401d87c9 2024-03-26T02:00:46.405 INFO:teuthology.orchestra.run.smithi019.stdout:osd.6 smithi069 running (9m) 48s ago 9m 345M 2704M 17.2.0 e1d6a67b021e b123f75154c8 2024-03-26T02:00:46.405 INFO:teuthology.orchestra.run.smithi019.stdout:osd.7 smithi069 running (9m) 48s ago 9m 550M 2704M 17.2.0 e1d6a67b021e d175fbc9ff8a 2024-03-26T02:00:46.405 INFO:teuthology.orchestra.run.smithi019.stdout:prometheus.smithi019 smithi019 *:9095 running (2m) 11s ago 12m 59.3M - 2.33.4 514e6a882f6e 29f89db72a07 2024-03-26T02:00:46.941 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:00:46.942 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": { 2024-03-26T02:00:46.942 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:00:46.942 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:00:46.942 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": { 2024-03-26T02:00:46.942 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:00:46.942 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:00:46.942 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": { 2024-03-26T02:00:46.942 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-03-26T02:00:46.942 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 1 2024-03-26T02:00:46.942 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:00:46.942 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": { 2024-03-26T02:00:46.942 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:00:46.942 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:00:46.943 INFO:teuthology.orchestra.run.smithi019.stdout: "overall": { 2024-03-26T02:00:46.943 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-03-26T02:00:46.943 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 5 2024-03-26T02:00:46.943 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-26T02:00:46.943 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:00:47.103 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:46 smithi019 ceph-mon[66359]: from='client.34214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:00:47.104 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:46 smithi019 ceph-mon[66359]: failed to encode map e85 with expected crc 2024-03-26T02:00:47.104 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:46 smithi019 ceph-mon[66359]: failed to encode map e85 with expected crc 2024-03-26T02:00:47.104 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:46 smithi019 ceph-mon[66359]: failed to encode map e85 with expected crc 2024-03-26T02:00:47.104 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:46 smithi019 ceph-mon[66359]: failed to encode map e85 with expected crc 2024-03-26T02:00:47.104 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:46 smithi019 ceph-mon[66359]: failed to encode map e85 with expected crc 2024-03-26T02:00:47.104 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:46 smithi019 ceph-mon[66359]: from='client.34218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:00:47.104 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:46 smithi019 ceph-mon[66359]: failed to encode map e85 with expected crc 2024-03-26T02:00:47.104 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:46 smithi019 ceph-mon[66359]: failed to encode map e86 with expected crc 2024-03-26T02:00:47.104 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:46 smithi019 ceph-mon[66359]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T02:00:47.111 INFO:teuthology.orchestra.run.smithi019.stdout: 5 209270 11.30 MB/sec execute 320 sec latency 1020.051 ms 2024-03-26T02:00:47.362 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:46 smithi069 ceph-mon[53087]: from='client.34214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:00:47.362 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:46 smithi069 ceph-mon[53087]: failed to encode map e85 with expected crc 2024-03-26T02:00:47.362 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:46 smithi069 ceph-mon[53087]: failed to encode map e85 with expected crc 2024-03-26T02:00:47.362 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:46 smithi069 ceph-mon[53087]: failed to encode map e85 with expected crc 2024-03-26T02:00:47.362 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:46 smithi069 ceph-mon[53087]: failed to encode map e85 with expected crc 2024-03-26T02:00:47.362 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:46 smithi069 ceph-mon[53087]: failed to encode map e85 with expected crc 2024-03-26T02:00:47.362 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:46 smithi069 ceph-mon[53087]: from='client.34218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:00:47.362 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:46 smithi069 ceph-mon[53087]: failed to encode map e85 with expected crc 2024-03-26T02:00:47.362 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:46 smithi069 ceph-mon[53087]: failed to encode map e86 with expected crc 2024-03-26T02:00:47.362 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:46 smithi069 ceph-mon[53087]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T02:00:47.438 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:00:47.439 INFO:teuthology.orchestra.run.smithi019.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2", 2024-03-26T02:00:47.439 INFO:teuthology.orchestra.run.smithi019.stdout: "in_progress": true, 2024-03-26T02:00:47.439 INFO:teuthology.orchestra.run.smithi019.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T02:00:47.439 INFO:teuthology.orchestra.run.smithi019.stdout: "services_complete": [ 2024-03-26T02:00:47.439 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr", 2024-03-26T02:00:47.439 INFO:teuthology.orchestra.run.smithi019.stdout: "crash", 2024-03-26T02:00:47.439 INFO:teuthology.orchestra.run.smithi019.stdout: "mon" 2024-03-26T02:00:47.439 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-26T02:00:47.439 INFO:teuthology.orchestra.run.smithi019.stdout: "progress": "7/23 daemons upgraded", 2024-03-26T02:00:47.439 INFO:teuthology.orchestra.run.smithi019.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T02:00:47.439 INFO:teuthology.orchestra.run.smithi019.stdout: "is_paused": false 2024-03-26T02:00:47.439 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:00:47.948 INFO:teuthology.orchestra.run.smithi019.stdout:HEALTH_WARN Degraded data redundancy: 37/1596 objects degraded (2.318%), 3 pgs degraded 2024-03-26T02:00:47.948 INFO:teuthology.orchestra.run.smithi019.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 37/1596 objects degraded (2.318%), 3 pgs degraded 2024-03-26T02:00:47.948 INFO:teuthology.orchestra.run.smithi019.stdout: pg 3.1 is active+recovery_wait+undersized+degraded+remapped, acting [3,4] 2024-03-26T02:00:47.948 INFO:teuthology.orchestra.run.smithi019.stdout: pg 3.18 is active+recovering+undersized+degraded+remapped, acting [3,1] 2024-03-26T02:00:47.948 INFO:teuthology.orchestra.run.smithi019.stdout: pg 3.1b is active+recovering+undersized+degraded+remapped, acting [4,7] 2024-03-26T02:00:48.111 INFO:teuthology.orchestra.run.smithi019.stdout: 5 209481 11.27 MB/sec execute 321 sec latency 52.943 ms 2024-03-26T02:00:48.145 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:47 smithi019 ceph-mon[66359]: from='client.34222 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:00:48.145 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:47 smithi019 ceph-mon[66359]: pgmap v83: 97 pgs: 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 peering, 2 active+recovering+undersized+degraded+remapped, 88 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.1 MiB/s wr, 457 op/s; 37/1596 objects degraded (2.318%); 0 B/s, 35 objects/s recovering 2024-03-26T02:00:48.145 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:47 smithi019 ceph-mon[66359]: failed to encode map e86 with expected crc 2024-03-26T02:00:48.145 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:47 smithi019 ceph-mon[66359]: failed to encode map e86 with expected crc 2024-03-26T02:00:48.146 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:47 smithi019 ceph-mon[66359]: failed to encode map e86 with expected crc 2024-03-26T02:00:48.146 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:47 smithi019 ceph-mon[66359]: failed to encode map e86 with expected crc 2024-03-26T02:00:48.146 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:47 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/1168351940' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:00:48.146 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:47 smithi019 ceph-mon[66359]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T02:00:48.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:47 smithi069 ceph-mon[53087]: from='client.34222 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:00:48.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:47 smithi069 ceph-mon[53087]: pgmap v83: 97 pgs: 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 peering, 2 active+recovering+undersized+degraded+remapped, 88 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.1 MiB/s wr, 457 op/s; 37/1596 objects degraded (2.318%); 0 B/s, 35 objects/s recovering 2024-03-26T02:00:48.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:47 smithi069 ceph-mon[53087]: failed to encode map e86 with expected crc 2024-03-26T02:00:48.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:47 smithi069 ceph-mon[53087]: failed to encode map e86 with expected crc 2024-03-26T02:00:48.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:47 smithi069 ceph-mon[53087]: failed to encode map e86 with expected crc 2024-03-26T02:00:48.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:47 smithi069 ceph-mon[53087]: failed to encode map e86 with expected crc 2024-03-26T02:00:48.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:47 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/1168351940' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:00:48.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:47 smithi069 ceph-mon[53087]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T02:00:49.111 INFO:teuthology.orchestra.run.smithi019.stdout: 5 209760 11.26 MB/sec execute 322 sec latency 54.234 ms 2024-03-26T02:00:49.182 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:48 smithi019 ceph-mon[66359]: from='client.34230 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:00:49.183 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:48 smithi019 ceph-mon[66359]: failed to encode map e87 with expected crc 2024-03-26T02:00:49.183 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:48 smithi019 ceph-mon[66359]: failed to encode map e87 with expected crc 2024-03-26T02:00:49.183 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:48 smithi019 ceph-mon[66359]: failed to encode map e87 with expected crc 2024-03-26T02:00:49.183 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:48 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/2388986190' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:00:49.183 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:48 smithi019 ceph-mon[66359]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T02:00:49.254 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:48 smithi069 ceph-mon[53087]: from='client.34230 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:00:49.255 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:48 smithi069 ceph-mon[53087]: failed to encode map e87 with expected crc 2024-03-26T02:00:49.255 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:48 smithi069 ceph-mon[53087]: failed to encode map e87 with expected crc 2024-03-26T02:00:49.255 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:48 smithi069 ceph-mon[53087]: failed to encode map e87 with expected crc 2024-03-26T02:00:49.255 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:48 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/2388986190' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:00:49.255 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:48 smithi069 ceph-mon[53087]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T02:00:49.963 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:49 smithi019 ceph-mon[66359]: pgmap v86: 97 pgs: 1 active+undersized+remapped, 2 peering, 1 active+recovering+undersized+degraded+remapped, 93 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.5 KiB/s rd, 5.7 MiB/s wr, 492 op/s; 7/1632 objects degraded (0.429%); 0 B/s, 55 objects/s recovering 2024-03-26T02:00:49.963 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:49 smithi019 ceph-mon[66359]: failed to encode map e88 with expected crc 2024-03-26T02:00:49.964 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:49 smithi019 ceph-mon[66359]: failed to encode map e88 with expected crc 2024-03-26T02:00:49.964 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:49 smithi019 ceph-mon[66359]: failed to encode map e88 with expected crc 2024-03-26T02:00:49.964 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:49 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 7/1632 objects degraded (0.429%), 1 pg degraded (PG_DEGRADED) 2024-03-26T02:00:50.111 INFO:teuthology.orchestra.run.smithi019.stdout: 5 209964 11.25 MB/sec execute 323 sec latency 54.310 ms 2024-03-26T02:00:50.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:49 smithi069 ceph-mon[53087]: pgmap v86: 97 pgs: 1 active+undersized+remapped, 2 peering, 1 active+recovering+undersized+degraded+remapped, 93 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.5 KiB/s rd, 5.7 MiB/s wr, 492 op/s; 7/1632 objects degraded (0.429%); 0 B/s, 55 objects/s recovering 2024-03-26T02:00:50.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:49 smithi069 ceph-mon[53087]: failed to encode map e88 with expected crc 2024-03-26T02:00:50.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:49 smithi069 ceph-mon[53087]: failed to encode map e88 with expected crc 2024-03-26T02:00:50.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:49 smithi069 ceph-mon[53087]: failed to encode map e88 with expected crc 2024-03-26T02:00:50.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:49 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 7/1632 objects degraded (0.429%), 1 pg degraded (PG_DEGRADED) 2024-03-26T02:00:51.111 INFO:teuthology.orchestra.run.smithi019.stdout: 5 210235 11.24 MB/sec execute 324 sec latency 95.442 ms 2024-03-26T02:00:52.044 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:51 smithi019 ceph-mon[66359]: pgmap v88: 97 pgs: 1 active+undersized+remapped, 2 peering, 1 active+recovering+undersized+degraded+remapped, 93 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.8 KiB/s rd, 4.8 MiB/s wr, 412 op/s; 7/1632 objects degraded (0.429%); 0 B/s, 46 objects/s recovering 2024-03-26T02:00:52.111 INFO:teuthology.orchestra.run.smithi019.stdout: 5 210519 11.23 MB/sec execute 325 sec latency 85.229 ms 2024-03-26T02:00:52.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:51 smithi069 ceph-mon[53087]: pgmap v88: 97 pgs: 1 active+undersized+remapped, 2 peering, 1 active+recovering+undersized+degraded+remapped, 93 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.8 KiB/s rd, 4.8 MiB/s wr, 412 op/s; 7/1632 objects degraded (0.429%); 0 B/s, 46 objects/s recovering 2024-03-26T02:00:53.058 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:52 smithi019 ceph-mon[66359]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/1632 objects degraded (0.429%), 1 pg degraded) 2024-03-26T02:00:53.058 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:52 smithi019 ceph-mon[66359]: Cluster is now healthy 2024-03-26T02:00:53.111 INFO:teuthology.orchestra.run.smithi019.stdout: 5 210804 11.20 MB/sec execute 326 sec latency 83.635 ms 2024-03-26T02:00:53.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:52 smithi069 ceph-mon[53087]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/1632 objects degraded (0.429%), 1 pg degraded) 2024-03-26T02:00:53.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:52 smithi069 ceph-mon[53087]: Cluster is now healthy 2024-03-26T02:00:54.083 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:53 smithi019 ceph-mon[66359]: pgmap v89: 97 pgs: 97 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 3.2 MiB/s wr, 336 op/s; 0 B/s, 15 objects/s recovering 2024-03-26T02:00:54.112 INFO:teuthology.orchestra.run.smithi019.stdout: 5 211084 11.19 MB/sec execute 327 sec latency 57.419 ms 2024-03-26T02:00:54.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:53 smithi069 ceph-mon[53087]: pgmap v89: 97 pgs: 97 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 3.2 MiB/s wr, 336 op/s; 0 B/s, 15 objects/s recovering 2024-03-26T02:00:55.114 INFO:teuthology.orchestra.run.smithi019.stdout: 5 211354 11.18 MB/sec execute 328 sec latency 41.876 ms 2024-03-26T02:00:56.112 INFO:teuthology.orchestra.run.smithi019.stdout: 5 211664 11.19 MB/sec execute 329 sec latency 40.724 ms 2024-03-26T02:00:56.119 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:55 smithi019 ceph-mon[66359]: pgmap v90: 97 pgs: 97 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.6 KiB/s rd, 2.2 MiB/s wr, 241 op/s; 0 B/s, 2 objects/s recovering 2024-03-26T02:00:56.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:55 smithi069 ceph-mon[53087]: pgmap v90: 97 pgs: 97 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.6 KiB/s rd, 2.2 MiB/s wr, 241 op/s; 0 B/s, 2 objects/s recovering 2024-03-26T02:00:57.112 INFO:teuthology.orchestra.run.smithi019.stdout: 5 212034 11.18 MB/sec execute 330 sec latency 78.710 ms 2024-03-26T02:00:58.112 INFO:teuthology.orchestra.run.smithi019.stdout: 5 212354 11.18 MB/sec execute 331 sec latency 59.064 ms 2024-03-26T02:00:58.167 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:57 smithi019 ceph-mon[66359]: pgmap v91: 97 pgs: 97 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.1 KiB/s rd, 4.2 MiB/s wr, 440 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T02:00:58.168 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:57 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T02:00:58.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:57 smithi069 ceph-mon[53087]: pgmap v91: 97 pgs: 97 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.1 KiB/s rd, 4.2 MiB/s wr, 440 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T02:00:58.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:57 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T02:00:59.112 INFO:teuthology.orchestra.run.smithi019.stdout: 5 212672 11.17 MB/sec execute 332 sec latency 43.009 ms 2024-03-26T02:00:59.185 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:59 smithi019 ceph-mon[66359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T02:00:59.185 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:00:59 smithi019 ceph-mon[66359]: Upgrade: osd.1 is safe to restart 2024-03-26T02:00:59.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:59 smithi069 ceph-mon[53087]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T02:00:59.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:00:59 smithi069 ceph-mon[53087]: Upgrade: osd.1 is safe to restart 2024-03-26T02:01:00.112 INFO:teuthology.orchestra.run.smithi019.stdout: 5 212981 11.15 MB/sec execute 333 sec latency 42.238 ms 2024-03-26T02:01:00.194 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:00 smithi019 ceph-mon[66359]: pgmap v92: 97 pgs: 97 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 3.9 MiB/s wr, 401 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T02:01:00.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:00 smithi069 ceph-mon[53087]: pgmap v92: 97 pgs: 97 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 3.9 MiB/s wr, 401 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T02:01:01.112 INFO:teuthology.orchestra.run.smithi019.stdout: 5 213264 11.14 MB/sec execute 334 sec latency 54.412 ms 2024-03-26T02:01:01.692 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:01 smithi019 ceph-mon[66359]: Upgrade: Updating osd.1 2024-03-26T02:01:01.692 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:01.692 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T02:01:01.692 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:01:01.692 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:01 smithi019 ceph-mon[66359]: Deploying daemon osd.1 on smithi019 2024-03-26T02:01:01.692 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:01 smithi019 ceph-mon[66359]: pgmap v93: 97 pgs: 97 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.3 MiB/s wr, 341 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T02:01:01.692 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:01.692 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:01.693 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:01:01.915 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:01 smithi069 ceph-mon[53087]: Upgrade: Updating osd.1 2024-03-26T02:01:01.915 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:01.915 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T02:01:01.915 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:01:01.915 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:01 smithi069 ceph-mon[53087]: Deploying daemon osd.1 on smithi019 2024-03-26T02:01:01.915 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:01 smithi069 ceph-mon[53087]: pgmap v93: 97 pgs: 97 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.3 MiB/s wr, 341 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T02:01:01.915 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:01.915 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:01.915 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:01:02.112 INFO:teuthology.orchestra.run.smithi019.stdout: 5 213522 11.13 MB/sec execute 335 sec latency 60.856 ms 2024-03-26T02:01:03.112 INFO:teuthology.orchestra.run.smithi019.stdout: 5 213848 11.12 MB/sec execute 336 sec latency 50.595 ms 2024-03-26T02:01:04.000 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:03 smithi019 ceph-mon[66359]: pgmap v94: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.3 MiB/s wr, 531 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T02:01:04.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:03 smithi069 ceph-mon[53087]: pgmap v94: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.3 MiB/s wr, 531 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T02:01:04.112 INFO:teuthology.orchestra.run.smithi019.stdout: 5 214174 11.11 MB/sec execute 337 sec latency 52.643 ms 2024-03-26T02:01:05.112 INFO:teuthology.orchestra.run.smithi019.stdout: 5 214472 11.09 MB/sec execute 338 sec latency 82.067 ms 2024-03-26T02:01:05.785 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:05 smithi019 ceph-mon[66359]: pgmap v95: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.9 MiB/s wr, 375 op/s 2024-03-26T02:01:05.785 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:05 smithi019 systemd[1]: Stopping Ceph osd.1 for 692607c6-eb12-11ee-95d0-87774f69a715... 2024-03-26T02:01:06.037 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:06 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1[39358]: 2024-03-26T02:01:06.009+0000 7fc1ee4b3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:01:06.037 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:06 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1[39358]: 2024-03-26T02:01:06.009+0000 7fc1ee4b3700 -1 osd.1 88 *** Got signal Terminated *** 2024-03-26T02:01:06.037 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:06 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1[39358]: 2024-03-26T02:01:06.009+0000 7fc1ee4b3700 -1 osd.1 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:01:06.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:05 smithi069 ceph-mon[53087]: pgmap v95: 97 pgs: 97 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.9 MiB/s wr, 375 op/s 2024-03-26T02:01:06.112 INFO:teuthology.orchestra.run.smithi019.stdout: 5 214729 11.09 MB/sec execute 339 sec latency 36.406 ms 2024-03-26T02:01:06.802 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:06 smithi019 ceph-mon[66359]: osd.1 marked itself down and dead 2024-03-26T02:01:07.010 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:06 smithi069 ceph-mon[53087]: osd.1 marked itself down and dead 2024-03-26T02:01:07.112 INFO:teuthology.orchestra.run.smithi019.stdout: 5 215000 11.07 MB/sec execute 340 sec latency 317.756 ms 2024-03-26T02:01:07.678 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:07 smithi019 podman[74639]: 2024-03-26 02:01:07.424916191 +0000 UTC m=+1.649856284 container died 31d3489c71887820f51db78911325f7474b0a2a01d6dd65a761f20712c245661 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, GIT_CLEAN=True, architecture=x86_64, GIT_BRANCH=HEAD, ceph=True, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, distribution-scope=public, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, vendor=Red Hat, Inc., io.openshift.expose-services=, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, maintainer=Guillaume Abrioux ) 2024-03-26T02:01:07.939 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:07 smithi019 ceph-mon[66359]: pgmap v96: 97 pgs: 97 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.3 KiB/s rd, 5.4 MiB/s wr, 547 op/s 2024-03-26T02:01:07.940 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:07 smithi019 ceph-mon[66359]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:01:07.940 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:07 smithi019 ceph-mon[66359]: failed to encode map e89 with expected crc 2024-03-26T02:01:07.940 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:07 smithi019 ceph-mon[66359]: osdmap e89: 8 total, 7 up, 8 in 2024-03-26T02:01:07.940 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:07 smithi019 ceph-mon[66359]: failed to encode map e89 with expected crc 2024-03-26T02:01:07.940 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:07 smithi019 podman[74639]: 2024-03-26 02:01:07.76045026 +0000 UTC m=+1.985390339 container cleanup 31d3489c71887820f51db78911325f7474b0a2a01d6dd65a761f20712c245661 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.19.8, vendor=Red Hat, Inc., RELEASE=HEAD, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, name=centos-stream, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=) 2024-03-26T02:01:07.940 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:07 smithi019 bash[74639]: ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1 2024-03-26T02:01:08.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:07 smithi069 ceph-mon[53087]: pgmap v96: 97 pgs: 97 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.3 KiB/s rd, 5.4 MiB/s wr, 547 op/s 2024-03-26T02:01:08.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:07 smithi069 ceph-mon[53087]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:01:08.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:07 smithi069 ceph-mon[53087]: failed to encode map e89 with expected crc 2024-03-26T02:01:08.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:07 smithi069 ceph-mon[53087]: osdmap e89: 8 total, 7 up, 8 in 2024-03-26T02:01:08.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:07 smithi069 ceph-mon[53087]: failed to encode map e89 with expected crc 2024-03-26T02:01:08.112 INFO:teuthology.orchestra.run.smithi019.stdout: 5 215096 11.05 MB/sec execute 341 sec latency 996.216 ms 2024-03-26T02:01:08.194 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:08 smithi019 podman[74659]: 2024-03-26 02:01:08.054880859 +0000 UTC m=+0.631753871 container remove 31d3489c71887820f51db78911325f7474b0a2a01d6dd65a761f20712c245661 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, GIT_CLEAN=True, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, architecture=x86_64, com.redhat.component=centos-stream-container, io.openshift.expose-services=, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, distribution-scope=public, ceph=True, name=centos-stream, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8) 2024-03-26T02:01:08.747 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:08 smithi019 podman[74749]: 2024-03-26 02:01:08.466799918 +0000 UTC m=+0.018024348 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:01:08.747 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:08 smithi019 podman[74749]: 2024-03-26 02:01:08.659336804 +0000 UTC m=+0.210561233 container create e78e2f7b891d537ca4de697beb49d8fc77ee02ee953413d5923adeab1cc11b1f (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-26T02:01:09.005 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:08 smithi019 ceph-mon[66359]: failed to encode map e90 with expected crc 2024-03-26T02:01:09.006 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:08 smithi019 ceph-mon[66359]: osdmap e90: 8 total, 7 up, 8 in 2024-03-26T02:01:09.006 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:08 smithi019 ceph-mon[66359]: failed to encode map e90 with expected crc 2024-03-26T02:01:09.006 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:08 smithi019 ceph-mon[66359]: failed to encode map e90 with expected crc 2024-03-26T02:01:09.006 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:08 smithi019 ceph-mon[66359]: failed to encode map e90 with expected crc 2024-03-26T02:01:09.006 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:08 smithi019 ceph-mon[66359]: failed to encode map e90 with expected crc 2024-03-26T02:01:09.006 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:08 smithi019 ceph-mon[66359]: failed to encode map e90 with expected crc 2024-03-26T02:01:09.006 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:08 smithi019 ceph-mon[66359]: failed to encode map e90 with expected crc 2024-03-26T02:01:09.006 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:08 smithi019 ceph-mon[66359]: failed to encode map e90 with expected crc 2024-03-26T02:01:09.006 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:08 smithi019 ceph-mon[66359]: failed to encode map e90 with expected crc 2024-03-26T02:01:09.006 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:08 smithi019 ceph-mon[66359]: failed to encode map e90 with expected crc 2024-03-26T02:01:09.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:08 smithi069 ceph-mon[53087]: failed to encode map e90 with expected crc 2024-03-26T02:01:09.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:08 smithi069 ceph-mon[53087]: osdmap e90: 8 total, 7 up, 8 in 2024-03-26T02:01:09.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:08 smithi069 ceph-mon[53087]: failed to encode map e90 with expected crc 2024-03-26T02:01:09.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:08 smithi069 ceph-mon[53087]: failed to encode map e90 with expected crc 2024-03-26T02:01:09.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:08 smithi069 ceph-mon[53087]: failed to encode map e90 with expected crc 2024-03-26T02:01:09.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:08 smithi069 ceph-mon[53087]: failed to encode map e90 with expected crc 2024-03-26T02:01:09.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:08 smithi069 ceph-mon[53087]: failed to encode map e90 with expected crc 2024-03-26T02:01:09.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:08 smithi069 ceph-mon[53087]: failed to encode map e90 with expected crc 2024-03-26T02:01:09.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:08 smithi069 ceph-mon[53087]: failed to encode map e90 with expected crc 2024-03-26T02:01:09.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:08 smithi069 ceph-mon[53087]: failed to encode map e90 with expected crc 2024-03-26T02:01:09.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:08 smithi069 ceph-mon[53087]: failed to encode map e90 with expected crc 2024-03-26T02:01:09.112 INFO:teuthology.orchestra.run.smithi019.stdout: 5 215458 11.06 MB/sec execute 342 sec latency 74.781 ms 2024-03-26T02:01:09.257 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:09 smithi019 podman[74749]: 2024-03-26 02:01:09.036247387 +0000 UTC m=+0.587471839 container init e78e2f7b891d537ca4de697beb49d8fc77ee02ee953413d5923adeab1cc11b1f (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:01:09.258 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:09 smithi019 podman[74749]: 2024-03-26 02:01:09.047441156 +0000 UTC m=+0.598665589 container start e78e2f7b891d537ca4de697beb49d8fc77ee02ee953413d5923adeab1cc11b1f (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-26T02:01:09.429 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:09 smithi019 podman[74749]: 2024-03-26 02:01:09.286709897 +0000 UTC m=+0.837934337 container attach e78e2f7b891d537ca4de697beb49d8fc77ee02ee953413d5923adeab1cc11b1f (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1-deactivate, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=squid-5e70124) 2024-03-26T02:01:09.509 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:09 smithi019 podman[74749]: 2024-03-26 02:01:09.487671068 +0000 UTC m=+1.038895504 container died e78e2f7b891d537ca4de697beb49d8fc77ee02ee953413d5923adeab1cc11b1f (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.build-date=20240319) 2024-03-26T02:01:10.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:09 smithi019 ceph-mon[66359]: pgmap v99: 97 pgs: 4 peering, 12 stale+active+clean, 81 active+clean; 336 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.7 MiB/s wr, 576 op/s 2024-03-26T02:01:10.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:09 smithi069 ceph-mon[53087]: pgmap v99: 97 pgs: 4 peering, 12 stale+active+clean, 81 active+clean; 336 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.7 MiB/s wr, 576 op/s 2024-03-26T02:01:10.112 INFO:teuthology.orchestra.run.smithi019.stdout: 5 215889 11.07 MB/sec execute 343 sec latency 34.611 ms 2024-03-26T02:01:10.777 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:10 smithi019 podman[74774]: 2024-03-26 02:01:10.656056827 +0000 UTC m=+1.158736619 container remove e78e2f7b891d537ca4de697beb49d8fc77ee02ee953413d5923adeab1cc11b1f (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-26T02:01:10.778 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:10 smithi019 systemd[1]: ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.1.service: Deactivated successfully. 2024-03-26T02:01:10.778 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:10 smithi019 systemd[1]: Stopped Ceph osd.1 for 692607c6-eb12-11ee-95d0-87774f69a715. 2024-03-26T02:01:10.778 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:10 smithi019 systemd[1]: ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.1.service: Consumed 3min 52.635s CPU time. 2024-03-26T02:01:11.112 INFO:teuthology.orchestra.run.smithi019.stdout: 5 216284 11.06 MB/sec execute 344 sec latency 37.946 ms 2024-03-26T02:01:11.293 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:11 smithi019 systemd[1]: Starting Ceph osd.1 for 692607c6-eb12-11ee-95d0-87774f69a715... 2024-03-26T02:01:11.548 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:11 smithi019 podman[74867]: 2024-03-26 02:01:11.232056826 +0000 UTC m=+0.028626002 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:01:11.549 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:11 smithi019 podman[74867]: 2024-03-26 02:01:11.390444497 +0000 UTC m=+0.187013668 container create 7c2ce517c4473cc1fcdcde44ac6f7953dda9561c65ebaeb5186808686f0cb288 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-26T02:01:11.801 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:11 smithi019 podman[74867]: 2024-03-26 02:01:11.793165052 +0000 UTC m=+0.589734234 container init 7c2ce517c4473cc1fcdcde44ac6f7953dda9561c65ebaeb5186808686f0cb288 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1-activate, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:01:11.801 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:11 smithi019 ceph-mon[66359]: pgmap v100: 97 pgs: 4 peering, 12 stale+active+clean, 81 active+clean; 336 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.7 MiB/s wr, 281 op/s 2024-03-26T02:01:12.052 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:11 smithi019 podman[74867]: 2024-03-26 02:01:11.804860904 +0000 UTC m=+0.601430072 container start 7c2ce517c4473cc1fcdcde44ac6f7953dda9561c65ebaeb5186808686f0cb288 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS) 2024-03-26T02:01:12.052 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:11 smithi019 podman[74867]: 2024-03-26 02:01:11.982450267 +0000 UTC m=+0.779019434 container attach 7c2ce517c4473cc1fcdcde44ac6f7953dda9561c65ebaeb5186808686f0cb288 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1-activate, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:01:12.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:11 smithi069 ceph-mon[53087]: pgmap v100: 97 pgs: 4 peering, 12 stale+active+clean, 81 active+clean; 336 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.7 MiB/s wr, 281 op/s 2024-03-26T02:01:12.112 INFO:teuthology.orchestra.run.smithi019.stdout: 5 216730 11.05 MB/sec execute 345 sec latency 34.159 ms 2024-03-26T02:01:12.559 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:12 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1-activate[74881]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-26T02:01:12.559 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:12 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1-activate[74881]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-26T02:01:12.559 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:12 smithi019 bash[74867]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-26T02:01:12.559 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:12 smithi019 bash[74867]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-26T02:01:12.559 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:12 smithi019 bash[74867]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-26T02:01:12.559 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:12 smithi019 bash[74867]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-26T02:01:12.560 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:12 smithi019 bash[74867]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-26T02:01:12.560 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:12 smithi019 bash[74867]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-26T02:01:12.560 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:12 smithi019 bash[74867]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-26T02:01:12.560 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:12 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1-activate[74881]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-26T02:01:12.560 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:12 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1-activate[74881]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-26T02:01:12.560 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:12 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1-activate[74881]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-26T02:01:12.560 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:12 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1-activate[74881]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-26T02:01:12.560 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:12 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1-activate[74881]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-26T02:01:12.560 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:12 smithi019 podman[74867]: 2024-03-26 02:01:12.486964037 +0000 UTC m=+1.283533207 container died 7c2ce517c4473cc1fcdcde44ac6f7953dda9561c65ebaeb5186808686f0cb288 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-26T02:01:13.063 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:12 smithi019 ceph-mon[66359]: Health check failed: Degraded data redundancy: 189/1719 objects degraded (10.995%), 23 pgs degraded (PG_DEGRADED) 2024-03-26T02:01:13.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:12 smithi069 ceph-mon[53087]: Health check failed: Degraded data redundancy: 189/1719 objects degraded (10.995%), 23 pgs degraded (PG_DEGRADED) 2024-03-26T02:01:13.112 INFO:teuthology.orchestra.run.smithi019.stdout: 5 217183 11.06 MB/sec execute 346 sec latency 64.287 ms 2024-03-26T02:01:13.566 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:13 smithi019 podman[75013]: 2024-03-26 02:01:13.43499603 +0000 UTC m=+0.940227537 container remove 7c2ce517c4473cc1fcdcde44ac6f7953dda9561c65ebaeb5186808686f0cb288 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1-activate, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:01:13.827 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:13 smithi019 podman[75071]: 2024-03-26 02:01:13.626588276 +0000 UTC m=+0.030062611 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:01:13.828 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:13 smithi019 podman[75071]: 2024-03-26 02:01:13.744619408 +0000 UTC m=+0.148093743 container create 758caf4ed841dfa52bd3a8115b7b0108fc27e1ad2ae2beadd60247c45dc57d9e (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:01:14.078 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:13 smithi019 ceph-mon[66359]: pgmap v101: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.7 MiB/s wr, 572 op/s; 189/1719 objects degraded (10.995%) 2024-03-26T02:01:14.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:13 smithi069 ceph-mon[53087]: pgmap v101: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.7 MiB/s wr, 572 op/s; 189/1719 objects degraded (10.995%) 2024-03-26T02:01:14.112 INFO:teuthology.orchestra.run.smithi019.stdout: 5 217657 11.07 MB/sec execute 347 sec latency 56.399 ms 2024-03-26T02:01:14.332 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:14 smithi019 podman[75071]: 2024-03-26 02:01:14.090228738 +0000 UTC m=+0.493703088 container init 758caf4ed841dfa52bd3a8115b7b0108fc27e1ad2ae2beadd60247c45dc57d9e (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-26T02:01:14.332 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:14 smithi019 podman[75071]: 2024-03-26 02:01:14.102495922 +0000 UTC m=+0.505970264 container start 758caf4ed841dfa52bd3a8115b7b0108fc27e1ad2ae2beadd60247c45dc57d9e (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-26T02:01:14.332 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:14 smithi019 ceph-osd[75088]: -- 172.21.15.19:0/2474393648 <== mon.1 v2:172.21.15.69:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a39842f1e0 con 0x55a398400c00 2024-03-26T02:01:14.333 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:14 smithi019 bash[75071]: 758caf4ed841dfa52bd3a8115b7b0108fc27e1ad2ae2beadd60247c45dc57d9e 2024-03-26T02:01:14.333 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:14 smithi019 systemd[1]: Started Ceph osd.1 for 692607c6-eb12-11ee-95d0-87774f69a715. 2024-03-26T02:01:14.843 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:14 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1[75084]: 2024-03-26T02:01:14.689+0000 7f85d90937c0 -1 Falling back to public interface 2024-03-26T02:01:15.112 INFO:teuthology.orchestra.run.smithi019.stdout: 5 218004 11.05 MB/sec execute 348 sec latency 76.920 ms 2024-03-26T02:01:15.608 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:15 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:15.609 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:15 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:15.609 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:15 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:01:15.609 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:15 smithi019 ceph-mon[66359]: pgmap v102: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.4 MiB/s wr, 313 op/s; 189/1719 objects degraded (10.995%) 2024-03-26T02:01:15.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:15 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:15.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:15 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:15.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:15 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:01:15.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:15 smithi069 ceph-mon[53087]: pgmap v102: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.4 MiB/s wr, 313 op/s; 189/1719 objects degraded (10.995%) 2024-03-26T02:01:16.112 INFO:teuthology.orchestra.run.smithi019.stdout: 5 218329 11.04 MB/sec execute 349 sec latency 34.582 ms 2024-03-26T02:01:16.877 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:16 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:16.878 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:16 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:01:17.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:16 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:17.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:16 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:01:17.112 INFO:teuthology.orchestra.run.smithi019.stdout: 5 218784 11.06 MB/sec execute 350 sec latency 46.487 ms 2024-03-26T02:01:17.914 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:17 smithi019 ceph-mon[66359]: pgmap v103: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 338 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 2.8 KiB/s rd, 4.8 MiB/s wr, 445 op/s; 183/1647 objects degraded (11.111%) 2024-03-26T02:01:18.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:17 smithi069 ceph-mon[53087]: pgmap v103: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 338 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 2.8 KiB/s rd, 4.8 MiB/s wr, 445 op/s; 183/1647 objects degraded (11.111%) 2024-03-26T02:01:18.112 INFO:teuthology.orchestra.run.smithi019.stdout: 5 219187 11.06 MB/sec execute 351 sec latency 66.608 ms 2024-03-26T02:01:18.482 INFO:teuthology.orchestra.run.smithi019.stdout:true 2024-03-26T02:01:18.920 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:18 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 182/1626 objects degraded (11.193%), 23 pgs degraded (PG_DEGRADED) 2024-03-26T02:01:19.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:18 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 182/1626 objects degraded (11.193%), 23 pgs degraded (PG_DEGRADED) 2024-03-26T02:01:19.112 INFO:teuthology.orchestra.run.smithi019.stdout: 5 219501 11.06 MB/sec execute 352 sec latency 50.715 ms 2024-03-26T02:01:19.595 INFO:teuthology.orchestra.run.smithi019.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:01:19.595 INFO:teuthology.orchestra.run.smithi019.stdout:alertmanager.smithi019 smithi019 *:9093,9094 running (5m) 44s ago 14m 14.4M - 0.23.0 ba2b418f427c a6630841c2b2 2024-03-26T02:01:19.595 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi019 smithi019 running (93s) 44s ago 14m 6660k - 19.0.0-1775-g5e70124b 5f21c9264136 c5f91a47ddf2 2024-03-26T02:01:19.595 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi069 smithi069 running (86s) 81s ago 13m 6669k - 19.0.0-1775-g5e70124b 5f21c9264136 05128803b49e 2024-03-26T02:01:19.595 INFO:teuthology.orchestra.run.smithi019.stdout:grafana.smithi019 smithi019 *:3000 running (4m) 44s ago 14m 47.3M - 8.3.5 dad864ee21e9 a97dea3b5fec 2024-03-26T02:01:19.595 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi019.lmryom smithi019 running (8m) 44s ago 8m 38.1M - 17.2.0 e1d6a67b021e 91a7c2942715 2024-03-26T02:01:19.595 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi069.spgwgo smithi069 running (8m) 81s ago 8m 14.4M - 17.2.0 e1d6a67b021e 94d96338dbe1 2024-03-26T02:01:19.595 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi019.ixuvdm smithi019 *:8443,9283,8765 running (4m) 44s ago 15m 499M - 19.0.0-1775-g5e70124b 5f21c9264136 4460b22c7a56 2024-03-26T02:01:19.596 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi069.wpbewu smithi069 *:8443,9283,8765 running (3m) 81s ago 13m 451M - 19.0.0-1775-g5e70124b 5f21c9264136 380fd33842a8 2024-03-26T02:01:19.596 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi019 smithi019 running (3m) 44s ago 15m 47.6M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 c18934705774 2024-03-26T02:01:19.596 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi069 smithi069 running (2m) 81s ago 13m 35.6M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 a856bd263104 2024-03-26T02:01:19.596 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.0.0.smithi019.qbizxo smithi019 *:2049 running (8m) 44s ago 8m 284M - 4.0 e1d6a67b021e cd6628a9075d 2024-03-26T02:01:19.596 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.1.0.smithi069.uawdhc smithi069 *:2049 running (8m) 81s ago 8m 85.3M - 4.0 e1d6a67b021e 6ee3695e7a10 2024-03-26T02:01:19.596 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi019 smithi019 *:9100 running (4m) 44s ago 14m 10.2M - 1.3.1 1dbe0e931976 0d1d184bec63 2024-03-26T02:01:19.596 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi069 smithi069 *:9100 running (4m) 81s ago 13m 19.2M - 1.3.1 1dbe0e931976 c49a006f94e7 2024-03-26T02:01:19.596 INFO:teuthology.orchestra.run.smithi019.stdout:osd.0 smithi019 running (54s) 44s ago 12m 12.6M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 3f931cfe320a 2024-03-26T02:01:19.596 INFO:teuthology.orchestra.run.smithi019.stdout:osd.1 smithi019 starting - - - 1937M 2024-03-26T02:01:19.596 INFO:teuthology.orchestra.run.smithi019.stdout:osd.2 smithi019 running (11m) 44s ago 11m 483M 1937M 17.2.0 e1d6a67b021e f549115e7685 2024-03-26T02:01:19.596 INFO:teuthology.orchestra.run.smithi019.stdout:osd.3 smithi019 running (11m) 44s ago 11m 601M 1937M 17.2.0 e1d6a67b021e 154af68d03e1 2024-03-26T02:01:19.596 INFO:teuthology.orchestra.run.smithi019.stdout:osd.4 smithi069 running (11m) 81s ago 11m 556M 2704M 17.2.0 e1d6a67b021e 4fd5e853f8bf 2024-03-26T02:01:19.596 INFO:teuthology.orchestra.run.smithi019.stdout:osd.5 smithi069 running (10m) 81s ago 10m 433M 2704M 17.2.0 e1d6a67b021e 932e401d87c9 2024-03-26T02:01:19.596 INFO:teuthology.orchestra.run.smithi019.stdout:osd.6 smithi069 running (10m) 81s ago 10m 345M 2704M 17.2.0 e1d6a67b021e b123f75154c8 2024-03-26T02:01:19.597 INFO:teuthology.orchestra.run.smithi019.stdout:osd.7 smithi069 running (9m) 81s ago 9m 550M 2704M 17.2.0 e1d6a67b021e d175fbc9ff8a 2024-03-26T02:01:19.597 INFO:teuthology.orchestra.run.smithi019.stdout:prometheus.smithi019 smithi019 *:9095 running (3m) 44s ago 13m 59.3M - 2.33.4 514e6a882f6e 29f89db72a07 2024-03-26T02:01:19.926 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:19 smithi019 ceph-mon[66359]: from='client.44175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:19.926 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:19 smithi019 ceph-mon[66359]: pgmap v104: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 337 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.9 MiB/s wr, 484 op/s; 182/1626 objects degraded (11.193%) 2024-03-26T02:01:19.926 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:19 smithi019 ceph-mon[66359]: from='client.44179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:20.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:19 smithi069 ceph-mon[53087]: from='client.44175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:20.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:19 smithi069 ceph-mon[53087]: pgmap v104: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 337 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.9 MiB/s wr, 484 op/s; 182/1626 objects degraded (11.193%) 2024-03-26T02:01:20.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:19 smithi069 ceph-mon[53087]: from='client.44179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:20.112 INFO:teuthology.orchestra.run.smithi019.stdout: 5 219876 11.05 MB/sec execute 353 sec latency 56.271 ms 2024-03-26T02:01:20.280 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:01:20.280 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": { 2024-03-26T02:01:20.280 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:01:20.280 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:01:20.280 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": { 2024-03-26T02:01:20.280 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:01:20.280 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:01:20.280 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": { 2024-03-26T02:01:20.280 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-03-26T02:01:20.280 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 1 2024-03-26T02:01:20.280 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:01:20.281 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": { 2024-03-26T02:01:20.281 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:01:20.281 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:01:20.281 INFO:teuthology.orchestra.run.smithi019.stdout: "overall": { 2024-03-26T02:01:20.281 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-03-26T02:01:20.281 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 5 2024-03-26T02:01:20.281 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-26T02:01:20.281 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:01:20.866 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:01:20.867 INFO:teuthology.orchestra.run.smithi019.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2", 2024-03-26T02:01:20.867 INFO:teuthology.orchestra.run.smithi019.stdout: "in_progress": true, 2024-03-26T02:01:20.867 INFO:teuthology.orchestra.run.smithi019.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T02:01:20.867 INFO:teuthology.orchestra.run.smithi019.stdout: "services_complete": [ 2024-03-26T02:01:20.867 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr", 2024-03-26T02:01:20.867 INFO:teuthology.orchestra.run.smithi019.stdout: "crash", 2024-03-26T02:01:20.867 INFO:teuthology.orchestra.run.smithi019.stdout: "mon" 2024-03-26T02:01:20.867 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-26T02:01:20.867 INFO:teuthology.orchestra.run.smithi019.stdout: "progress": "7/23 daemons upgraded", 2024-03-26T02:01:20.867 INFO:teuthology.orchestra.run.smithi019.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T02:01:20.867 INFO:teuthology.orchestra.run.smithi019.stdout: "is_paused": false 2024-03-26T02:01:20.867 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:01:20.929 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:20 smithi019 ceph-mon[66359]: from='client.34248 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:20.930 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:20 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/2834023162' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:01:21.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:20 smithi069 ceph-mon[53087]: from='client.34248 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:21.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:20 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/2834023162' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:01:21.113 INFO:teuthology.orchestra.run.smithi019.stdout: 5 220330 11.05 MB/sec execute 354 sec latency 40.199 ms 2024-03-26T02:01:21.553 INFO:teuthology.orchestra.run.smithi019.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 182/1626 objects degraded (11.193%), 23 pgs degraded 2024-03-26T02:01:21.554 INFO:teuthology.orchestra.run.smithi019.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-26T02:01:21.554 INFO:teuthology.orchestra.run.smithi019.stdout: osd.1 (root=default,host=smithi019) is down 2024-03-26T02:01:21.554 INFO:teuthology.orchestra.run.smithi019.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 182/1626 objects degraded (11.193%), 23 pgs degraded 2024-03-26T02:01:21.554 INFO:teuthology.orchestra.run.smithi019.stdout: pg 2.0 is active+undersized+degraded, acting [7,0] 2024-03-26T02:01:21.554 INFO:teuthology.orchestra.run.smithi019.stdout: pg 2.2 is active+undersized+degraded, acting [5,6] 2024-03-26T02:01:21.554 INFO:teuthology.orchestra.run.smithi019.stdout: pg 2.4 is active+undersized+degraded, acting [0,7] 2024-03-26T02:01:21.554 INFO:teuthology.orchestra.run.smithi019.stdout: pg 2.6 is active+undersized+degraded, acting [6,4] 2024-03-26T02:01:21.554 INFO:teuthology.orchestra.run.smithi019.stdout: pg 2.8 is active+undersized+degraded, acting [7,0] 2024-03-26T02:01:21.554 INFO:teuthology.orchestra.run.smithi019.stdout: pg 2.9 is active+undersized+degraded, acting [7,3] 2024-03-26T02:01:21.554 INFO:teuthology.orchestra.run.smithi019.stdout: pg 2.a is active+undersized+degraded, acting [3,7] 2024-03-26T02:01:21.554 INFO:teuthology.orchestra.run.smithi019.stdout: pg 2.d is active+undersized+degraded, acting [4,3] 2024-03-26T02:01:21.554 INFO:teuthology.orchestra.run.smithi019.stdout: pg 2.10 is active+undersized+degraded, acting [2,0] 2024-03-26T02:01:21.555 INFO:teuthology.orchestra.run.smithi019.stdout: pg 2.11 is active+undersized+degraded, acting [6,4] 2024-03-26T02:01:21.555 INFO:teuthology.orchestra.run.smithi019.stdout: pg 2.15 is active+undersized+degraded, acting [5,0] 2024-03-26T02:01:21.555 INFO:teuthology.orchestra.run.smithi019.stdout: pg 3.0 is active+undersized+degraded, acting [2,6] 2024-03-26T02:01:21.555 INFO:teuthology.orchestra.run.smithi019.stdout: pg 3.4 is active+undersized+degraded, acting [2,5] 2024-03-26T02:01:21.555 INFO:teuthology.orchestra.run.smithi019.stdout: pg 3.6 is active+undersized+degraded, acting [0,4] 2024-03-26T02:01:21.555 INFO:teuthology.orchestra.run.smithi019.stdout: pg 3.8 is active+undersized+degraded, acting [3,7] 2024-03-26T02:01:21.555 INFO:teuthology.orchestra.run.smithi019.stdout: pg 3.a is active+undersized+degraded, acting [6,4] 2024-03-26T02:01:21.555 INFO:teuthology.orchestra.run.smithi019.stdout: pg 3.e is active+undersized+degraded, acting [7,4] 2024-03-26T02:01:21.555 INFO:teuthology.orchestra.run.smithi019.stdout: pg 3.16 is active+undersized+degraded, acting [5,7] 2024-03-26T02:01:21.555 INFO:teuthology.orchestra.run.smithi019.stdout: pg 3.18 is active+undersized+degraded, acting [3,0] 2024-03-26T02:01:21.555 INFO:teuthology.orchestra.run.smithi019.stdout: pg 3.19 is active+undersized+degraded, acting [3,4] 2024-03-26T02:01:21.555 INFO:teuthology.orchestra.run.smithi019.stdout: pg 3.1a is active+undersized+degraded, acting [4,2] 2024-03-26T02:01:21.555 INFO:teuthology.orchestra.run.smithi019.stdout: pg 3.1c is active+undersized+degraded, acting [5,4] 2024-03-26T02:01:21.555 INFO:teuthology.orchestra.run.smithi019.stdout: pg 4.14 is active+undersized+degraded, acting [3,7] 2024-03-26T02:01:21.938 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:21 smithi019 ceph-mon[66359]: pgmap v105: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 337 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.5 MiB/s wr, 441 op/s; 182/1626 objects degraded (11.193%) 2024-03-26T02:01:21.939 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:21 smithi019 ceph-mon[66359]: from='client.44191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:21.939 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:21 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/4026799801' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:01:22.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:21 smithi069 ceph-mon[53087]: pgmap v105: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 337 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.5 MiB/s wr, 441 op/s; 182/1626 objects degraded (11.193%) 2024-03-26T02:01:22.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:21 smithi069 ceph-mon[53087]: from='client.44191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:22.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:21 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/4026799801' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:01:22.112 INFO:teuthology.orchestra.run.smithi019.stdout: 5 220760 11.05 MB/sec execute 355 sec latency 32.439 ms 2024-03-26T02:01:22.443 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:22 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1[75084]: 2024-03-26T02:01:22.290+0000 7f85d90937c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-03-26T02:01:23.113 INFO:teuthology.orchestra.run.smithi019.stdout: 5 221182 11.05 MB/sec execute 356 sec latency 53.134 ms 2024-03-26T02:01:23.960 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:23 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1[75084]: 2024-03-26T02:01:23.726+0000 7f85d90937c0 -1 osd.1 88 log_to_monitors true 2024-03-26T02:01:23.961 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:23 smithi019 ceph-mon[66359]: pgmap v106: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 346 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.3 KiB/s rd, 7.0 MiB/s wr, 683 op/s; 205/1800 objects degraded (11.389%) 2024-03-26T02:01:23.961 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:23.961 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:23.961 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:23 smithi019 ceph-mon[66359]: from='osd.1 [v2:172.21.15.19:6810/216362633,v1:172.21.15.19:6811/216362633]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T02:01:24.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:23 smithi069 ceph-mon[53087]: pgmap v106: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 346 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.3 KiB/s rd, 7.0 MiB/s wr, 683 op/s; 205/1800 objects degraded (11.389%) 2024-03-26T02:01:24.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:24.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:24.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:23 smithi069 ceph-mon[53087]: from='osd.1 [v2:172.21.15.19:6810/216362633,v1:172.21.15.19:6811/216362633]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T02:01:24.112 INFO:teuthology.orchestra.run.smithi019.stdout: 5 221534 11.04 MB/sec execute 357 sec latency 79.667 ms 2024-03-26T02:01:24.727 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:01:24 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1[75084]: 2024-03-26T02:01:24.675+0000 7f85ca4eb700 -1 osd.1 88 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:01:24.979 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:24 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 205/1800 objects degraded (11.389%), 23 pgs degraded (PG_DEGRADED) 2024-03-26T02:01:24.980 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:24 smithi019 ceph-mon[66359]: from='osd.1 [v2:172.21.15.19:6810/216362633,v1:172.21.15.19:6811/216362633]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T02:01:24.980 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:24 smithi019 ceph-mon[66359]: osdmap e91: 8 total, 7 up, 8 in 2024-03-26T02:01:24.980 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:24 smithi019 ceph-mon[66359]: from='osd.1 [v2:172.21.15.19:6810/216362633,v1:172.21.15.19:6811/216362633]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-03-26T02:01:25.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:24 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 205/1800 objects degraded (11.389%), 23 pgs degraded (PG_DEGRADED) 2024-03-26T02:01:25.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:24 smithi069 ceph-mon[53087]: from='osd.1 [v2:172.21.15.19:6810/216362633,v1:172.21.15.19:6811/216362633]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T02:01:25.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:24 smithi069 ceph-mon[53087]: osdmap e91: 8 total, 7 up, 8 in 2024-03-26T02:01:25.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:24 smithi069 ceph-mon[53087]: from='osd.1 [v2:172.21.15.19:6810/216362633,v1:172.21.15.19:6811/216362633]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-03-26T02:01:25.113 INFO:teuthology.orchestra.run.smithi019.stdout: 5 221836 11.03 MB/sec execute 358 sec latency 36.034 ms 2024-03-26T02:01:26.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:25 smithi069 ceph-mon[53087]: failed to encode map e91 with expected crc 2024-03-26T02:01:26.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:25 smithi069 ceph-mon[53087]: failed to encode map e91 with expected crc 2024-03-26T02:01:26.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:25 smithi069 ceph-mon[53087]: pgmap v108: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.9 MiB/s wr, 587 op/s; 205/1800 objects degraded (11.389%) 2024-03-26T02:01:26.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:25 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:26.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:25 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:26.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:25 smithi069 ceph-mon[53087]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:01:26.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:25 smithi069 ceph-mon[53087]: osd.1 [v2:172.21.15.19:6810/216362633,v1:172.21.15.19:6811/216362633] boot 2024-03-26T02:01:26.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:25 smithi069 ceph-mon[53087]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T02:01:26.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:25 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:01:26.113 INFO:teuthology.orchestra.run.smithi019.stdout: 5 222096 11.02 MB/sec execute 359 sec latency 489.762 ms 2024-03-26T02:01:26.131 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:25 smithi019 ceph-mon[66359]: failed to encode map e91 with expected crc 2024-03-26T02:01:26.131 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:25 smithi019 ceph-mon[66359]: failed to encode map e91 with expected crc 2024-03-26T02:01:26.131 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:25 smithi019 ceph-mon[66359]: pgmap v108: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.9 MiB/s wr, 587 op/s; 205/1800 objects degraded (11.389%) 2024-03-26T02:01:26.131 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:25 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:26.131 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:25 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:26.131 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:25 smithi019 ceph-mon[66359]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:01:26.131 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:25 smithi019 ceph-mon[66359]: osd.1 [v2:172.21.15.19:6810/216362633,v1:172.21.15.19:6811/216362633] boot 2024-03-26T02:01:26.131 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:25 smithi019 ceph-mon[66359]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T02:01:26.131 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:25 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:01:27.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:26 smithi069 ceph-mon[53087]: failed to encode map e92 with expected crc 2024-03-26T02:01:27.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:26 smithi069 ceph-mon[53087]: osdmap e93: 8 total, 8 up, 8 in 2024-03-26T02:01:27.113 INFO:teuthology.orchestra.run.smithi019.stdout: 5 222096 10.99 MB/sec execute 360 sec latency 1489.807 ms 2024-03-26T02:01:27.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:26 smithi019 ceph-mon[66359]: failed to encode map e92 with expected crc 2024-03-26T02:01:27.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:26 smithi019 ceph-mon[66359]: osdmap e93: 8 total, 8 up, 8 in 2024-03-26T02:01:27.844 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:27 smithi019 ceph-mon[66359]: failed to encode map e93 with expected crc 2024-03-26T02:01:27.844 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:27 smithi019 ceph-mon[66359]: failed to encode map e93 with expected crc 2024-03-26T02:01:27.844 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:27 smithi019 ceph-mon[66359]: failed to encode map e93 with expected crc 2024-03-26T02:01:27.844 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:27 smithi019 ceph-mon[66359]: failed to encode map e93 with expected crc 2024-03-26T02:01:27.844 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:27 smithi019 ceph-mon[66359]: failed to encode map e93 with expected crc 2024-03-26T02:01:27.844 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:27 smithi019 ceph-mon[66359]: failed to encode map e93 with expected crc 2024-03-26T02:01:27.845 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:27 smithi019 ceph-mon[66359]: failed to encode map e93 with expected crc 2024-03-26T02:01:28.057 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:27 smithi019 ceph-mon[66359]: failed to encode map e93 with expected crc 2024-03-26T02:01:28.057 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:27 smithi019 ceph-mon[66359]: failed to encode map e93 with expected crc 2024-03-26T02:01:28.057 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:27 smithi019 ceph-mon[66359]: failed to encode map e93 with expected crc 2024-03-26T02:01:28.058 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:27 smithi019 ceph-mon[66359]: failed to encode map e93 with expected crc 2024-03-26T02:01:28.058 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:27 smithi019 ceph-mon[66359]: pgmap v111: 97 pgs: 2 activating+degraded, 8 remapped+peering, 2 activating, 10 active+undersized+degraded, 7 active+undersized, 68 active+clean; 340 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 8.0 MiB/s wr, 821 op/s; 99/1623 objects degraded (6.100%) 2024-03-26T02:01:28.058 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:27 smithi019 ceph-mon[66359]: osdmap e94: 8 total, 8 up, 8 in 2024-03-26T02:01:28.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:27 smithi069 ceph-mon[53087]: failed to encode map e93 with expected crc 2024-03-26T02:01:28.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:27 smithi069 ceph-mon[53087]: failed to encode map e93 with expected crc 2024-03-26T02:01:28.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:27 smithi069 ceph-mon[53087]: failed to encode map e93 with expected crc 2024-03-26T02:01:28.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:27 smithi069 ceph-mon[53087]: failed to encode map e93 with expected crc 2024-03-26T02:01:28.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:27 smithi069 ceph-mon[53087]: failed to encode map e93 with expected crc 2024-03-26T02:01:28.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:27 smithi069 ceph-mon[53087]: failed to encode map e93 with expected crc 2024-03-26T02:01:28.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:27 smithi069 ceph-mon[53087]: failed to encode map e93 with expected crc 2024-03-26T02:01:28.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:27 smithi069 ceph-mon[53087]: failed to encode map e93 with expected crc 2024-03-26T02:01:28.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:27 smithi069 ceph-mon[53087]: failed to encode map e93 with expected crc 2024-03-26T02:01:28.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:27 smithi069 ceph-mon[53087]: failed to encode map e93 with expected crc 2024-03-26T02:01:28.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:27 smithi069 ceph-mon[53087]: failed to encode map e93 with expected crc 2024-03-26T02:01:28.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:27 smithi069 ceph-mon[53087]: pgmap v111: 97 pgs: 2 activating+degraded, 8 remapped+peering, 2 activating, 10 active+undersized+degraded, 7 active+undersized, 68 active+clean; 340 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 8.0 MiB/s wr, 821 op/s; 99/1623 objects degraded (6.100%) 2024-03-26T02:01:28.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:27 smithi069 ceph-mon[53087]: osdmap e94: 8 total, 8 up, 8 in 2024-03-26T02:01:28.113 INFO:teuthology.orchestra.run.smithi019.stdout: 5 222222 10.98 MB/sec execute 361 sec latency 2213.138 ms 2024-03-26T02:01:29.098 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:28 smithi019 ceph-mon[66359]: failed to encode map e94 with expected crc 2024-03-26T02:01:29.099 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:28 smithi019 ceph-mon[66359]: failed to encode map e94 with expected crc 2024-03-26T02:01:29.099 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:28 smithi019 ceph-mon[66359]: failed to encode map e94 with expected crc 2024-03-26T02:01:29.099 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:28 smithi019 ceph-mon[66359]: failed to encode map e94 with expected crc 2024-03-26T02:01:29.099 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:28 smithi019 ceph-mon[66359]: failed to encode map e94 with expected crc 2024-03-26T02:01:29.099 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:28 smithi019 ceph-mon[66359]: failed to encode map e94 with expected crc 2024-03-26T02:01:29.099 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:28 smithi019 ceph-mon[66359]: failed to encode map e94 with expected crc 2024-03-26T02:01:29.099 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:28 smithi019 ceph-mon[66359]: failed to encode map e94 with expected crc 2024-03-26T02:01:29.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:28 smithi069 ceph-mon[53087]: failed to encode map e94 with expected crc 2024-03-26T02:01:29.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:28 smithi069 ceph-mon[53087]: failed to encode map e94 with expected crc 2024-03-26T02:01:29.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:28 smithi069 ceph-mon[53087]: failed to encode map e94 with expected crc 2024-03-26T02:01:29.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:28 smithi069 ceph-mon[53087]: failed to encode map e94 with expected crc 2024-03-26T02:01:29.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:28 smithi069 ceph-mon[53087]: failed to encode map e94 with expected crc 2024-03-26T02:01:29.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:28 smithi069 ceph-mon[53087]: failed to encode map e94 with expected crc 2024-03-26T02:01:29.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:28 smithi069 ceph-mon[53087]: failed to encode map e94 with expected crc 2024-03-26T02:01:29.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:28 smithi069 ceph-mon[53087]: failed to encode map e94 with expected crc 2024-03-26T02:01:29.113 INFO:teuthology.orchestra.run.smithi019.stdout: 5 222608 10.98 MB/sec execute 362 sec latency 37.908 ms 2024-03-26T02:01:30.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:29 smithi069 ceph-mon[53087]: pgmap v113: 97 pgs: 3 activating+degraded, 11 remapped+peering, 2 activating, 6 active+undersized+degraded, 4 active+undersized, 71 active+clean; 337 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 5.1 MiB/s wr, 549 op/s; 53/1587 objects degraded (3.340%) 2024-03-26T02:01:30.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:29 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 53/1587 objects degraded (3.340%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T02:01:30.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:29 smithi069 ceph-mon[53087]: osdmap e95: 8 total, 8 up, 8 in 2024-03-26T02:01:30.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:29 smithi069 ceph-mon[53087]: failed to encode map e95 with expected crc 2024-03-26T02:01:30.113 INFO:teuthology.orchestra.run.smithi019.stdout: 5 222707 10.96 MB/sec execute 363 sec latency 778.797 ms 2024-03-26T02:01:30.153 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:29 smithi019 ceph-mon[66359]: pgmap v113: 97 pgs: 3 activating+degraded, 11 remapped+peering, 2 activating, 6 active+undersized+degraded, 4 active+undersized, 71 active+clean; 337 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 5.1 MiB/s wr, 549 op/s; 53/1587 objects degraded (3.340%) 2024-03-26T02:01:30.154 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:29 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 53/1587 objects degraded (3.340%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T02:01:30.154 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:29 smithi019 ceph-mon[66359]: osdmap e95: 8 total, 8 up, 8 in 2024-03-26T02:01:30.154 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:29 smithi019 ceph-mon[66359]: failed to encode map e95 with expected crc 2024-03-26T02:01:30.919 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:30 smithi019 ceph-mon[66359]: failed to encode map e95 with expected crc 2024-03-26T02:01:30.919 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:30 smithi019 ceph-mon[66359]: failed to encode map e95 with expected crc 2024-03-26T02:01:30.919 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:30 smithi019 ceph-mon[66359]: failed to encode map e95 with expected crc 2024-03-26T02:01:30.919 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:30 smithi019 ceph-mon[66359]: failed to encode map e95 with expected crc 2024-03-26T02:01:30.919 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:30 smithi019 ceph-mon[66359]: failed to encode map e95 with expected crc 2024-03-26T02:01:30.919 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:30 smithi019 ceph-mon[66359]: failed to encode map e95 with expected crc 2024-03-26T02:01:30.919 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:30.919 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:30.919 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:01:30.919 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:01:30.919 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:30.919 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:30.920 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:01:30.920 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:01:30.920 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:01:30.920 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:01:30.920 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T02:01:30.920 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:30 smithi019 ceph-mon[66359]: osdmap e96: 8 total, 8 up, 8 in 2024-03-26T02:01:30.920 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:30 smithi019 ceph-mon[66359]: failed to encode map e96 with expected crc 2024-03-26T02:01:30.920 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:30.920 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:01:31.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:30 smithi069 ceph-mon[53087]: failed to encode map e95 with expected crc 2024-03-26T02:01:31.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:30 smithi069 ceph-mon[53087]: failed to encode map e95 with expected crc 2024-03-26T02:01:31.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:30 smithi069 ceph-mon[53087]: failed to encode map e95 with expected crc 2024-03-26T02:01:31.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:30 smithi069 ceph-mon[53087]: failed to encode map e95 with expected crc 2024-03-26T02:01:31.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:30 smithi069 ceph-mon[53087]: failed to encode map e95 with expected crc 2024-03-26T02:01:31.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:30 smithi069 ceph-mon[53087]: failed to encode map e95 with expected crc 2024-03-26T02:01:31.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:31.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:31.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:01:31.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:01:31.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:31.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:31.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:01:31.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:01:31.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:01:31.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:01:31.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T02:01:31.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:30 smithi069 ceph-mon[53087]: osdmap e96: 8 total, 8 up, 8 in 2024-03-26T02:01:31.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:30 smithi069 ceph-mon[53087]: failed to encode map e96 with expected crc 2024-03-26T02:01:31.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:31.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:01:31.113 INFO:teuthology.orchestra.run.smithi019.stdout: 5 222964 10.95 MB/sec execute 364 sec latency 1041.905 ms 2024-03-26T02:01:31.929 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:31 smithi019 ceph-mon[66359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T02:01:31.930 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:31 smithi019 ceph-mon[66359]: Upgrade: unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-03-26T02:01:31.930 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:31 smithi019 ceph-mon[66359]: failed to encode map e96 with expected crc 2024-03-26T02:01:31.930 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:31 smithi019 ceph-mon[66359]: failed to encode map e96 with expected crc 2024-03-26T02:01:31.930 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:31 smithi019 ceph-mon[66359]: failed to encode map e96 with expected crc 2024-03-26T02:01:31.930 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:31 smithi019 ceph-mon[66359]: failed to encode map e96 with expected crc 2024-03-26T02:01:31.930 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:31 smithi019 ceph-mon[66359]: failed to encode map e96 with expected crc 2024-03-26T02:01:31.930 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:31 smithi019 ceph-mon[66359]: pgmap v116: 97 pgs: 1 peering, 81 active+clean, 2 activating, 10 remapped+peering, 3 activating+degraded; 338 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 5.4 MiB/s wr, 563 op/s; 21/1584 objects degraded (1.326%); 22 B/s, 69 keys/s, 6 objects/s recovering 2024-03-26T02:01:31.930 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:31 smithi019 ceph-mon[66359]: osdmap e97: 8 total, 8 up, 8 in 2024-03-26T02:01:31.930 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:31 smithi019 ceph-mon[66359]: failed to encode map e97 with expected crc 2024-03-26T02:01:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:31 smithi069 ceph-mon[53087]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T02:01:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:31 smithi069 ceph-mon[53087]: Upgrade: unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-03-26T02:01:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:31 smithi069 ceph-mon[53087]: failed to encode map e96 with expected crc 2024-03-26T02:01:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:31 smithi069 ceph-mon[53087]: failed to encode map e96 with expected crc 2024-03-26T02:01:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:31 smithi069 ceph-mon[53087]: failed to encode map e96 with expected crc 2024-03-26T02:01:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:31 smithi069 ceph-mon[53087]: failed to encode map e96 with expected crc 2024-03-26T02:01:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:31 smithi069 ceph-mon[53087]: failed to encode map e96 with expected crc 2024-03-26T02:01:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:31 smithi069 ceph-mon[53087]: pgmap v116: 97 pgs: 1 peering, 81 active+clean, 2 activating, 10 remapped+peering, 3 activating+degraded; 338 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 5.4 MiB/s wr, 563 op/s; 21/1584 objects degraded (1.326%); 22 B/s, 69 keys/s, 6 objects/s recovering 2024-03-26T02:01:32.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:31 smithi069 ceph-mon[53087]: osdmap e97: 8 total, 8 up, 8 in 2024-03-26T02:01:32.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:31 smithi069 ceph-mon[53087]: failed to encode map e97 with expected crc 2024-03-26T02:01:32.113 INFO:teuthology.orchestra.run.smithi019.stdout: 5 223368 10.94 MB/sec execute 365 sec latency 34.607 ms 2024-03-26T02:01:32.933 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:32 smithi019 ceph-mon[66359]: failed to encode map e97 with expected crc 2024-03-26T02:01:32.933 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:32 smithi019 ceph-mon[66359]: failed to encode map e97 with expected crc 2024-03-26T02:01:32.934 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:32 smithi019 ceph-mon[66359]: osdmap e98: 8 total, 8 up, 8 in 2024-03-26T02:01:33.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:32 smithi069 ceph-mon[53087]: failed to encode map e97 with expected crc 2024-03-26T02:01:33.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:32 smithi069 ceph-mon[53087]: failed to encode map e97 with expected crc 2024-03-26T02:01:33.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:32 smithi069 ceph-mon[53087]: osdmap e98: 8 total, 8 up, 8 in 2024-03-26T02:01:33.113 INFO:teuthology.orchestra.run.smithi019.stdout: 5 223697 10.93 MB/sec execute 366 sec latency 761.467 ms 2024-03-26T02:01:33.943 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:33 smithi019 ceph-mon[66359]: failed to encode map e98 with expected crc 2024-03-26T02:01:33.944 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:33 smithi019 ceph-mon[66359]: failed to encode map e98 with expected crc 2024-03-26T02:01:33.944 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:33 smithi019 ceph-mon[66359]: failed to encode map e98 with expected crc 2024-03-26T02:01:33.944 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:33 smithi019 ceph-mon[66359]: pgmap v119: 97 pgs: 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 peering, 89 active+clean; 338 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.3 MiB/s wr, 400 op/s; 114/1617 objects degraded (7.050%); 643 KiB/s, 70 keys/s, 39 objects/s recovering 2024-03-26T02:01:33.944 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:33 smithi019 ceph-mon[66359]: osdmap e99: 8 total, 8 up, 8 in 2024-03-26T02:01:33.944 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:33 smithi019 ceph-mon[66359]: failed to encode map e99 with expected crc 2024-03-26T02:01:33.944 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:33 smithi019 ceph-mon[66359]: failed to encode map e99 with expected crc 2024-03-26T02:01:34.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:33 smithi069 ceph-mon[53087]: failed to encode map e98 with expected crc 2024-03-26T02:01:34.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:33 smithi069 ceph-mon[53087]: failed to encode map e98 with expected crc 2024-03-26T02:01:34.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:33 smithi069 ceph-mon[53087]: failed to encode map e98 with expected crc 2024-03-26T02:01:34.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:33 smithi069 ceph-mon[53087]: pgmap v119: 97 pgs: 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 peering, 89 active+clean; 338 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.3 MiB/s wr, 400 op/s; 114/1617 objects degraded (7.050%); 643 KiB/s, 70 keys/s, 39 objects/s recovering 2024-03-26T02:01:34.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:33 smithi069 ceph-mon[53087]: osdmap e99: 8 total, 8 up, 8 in 2024-03-26T02:01:34.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:33 smithi069 ceph-mon[53087]: failed to encode map e99 with expected crc 2024-03-26T02:01:34.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:33 smithi069 ceph-mon[53087]: failed to encode map e99 with expected crc 2024-03-26T02:01:34.113 INFO:teuthology.orchestra.run.smithi019.stdout: 5 224021 10.93 MB/sec execute 367 sec latency 1030.221 ms 2024-03-26T02:01:34.997 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:34 smithi019 ceph-mon[66359]: failed to encode map e99 with expected crc 2024-03-26T02:01:34.998 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:34 smithi019 ceph-mon[66359]: failed to encode map e99 with expected crc 2024-03-26T02:01:34.998 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:34 smithi019 ceph-mon[66359]: failed to encode map e99 with expected crc 2024-03-26T02:01:34.998 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:34 smithi019 ceph-mon[66359]: failed to encode map e99 with expected crc 2024-03-26T02:01:34.998 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:34 smithi019 ceph-mon[66359]: failed to encode map e99 with expected crc 2024-03-26T02:01:34.998 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:34 smithi019 ceph-mon[66359]: failed to encode map e99 with expected crc 2024-03-26T02:01:34.998 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:34 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 114/1617 objects degraded (7.050%), 6 pgs degraded (PG_DEGRADED) 2024-03-26T02:01:34.998 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:34 smithi019 ceph-mon[66359]: osdmap e100: 8 total, 8 up, 8 in 2024-03-26T02:01:34.998 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:34 smithi019 ceph-mon[66359]: failed to encode map e100 with expected crc 2024-03-26T02:01:35.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:34 smithi069 ceph-mon[53087]: failed to encode map e99 with expected crc 2024-03-26T02:01:35.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:34 smithi069 ceph-mon[53087]: failed to encode map e99 with expected crc 2024-03-26T02:01:35.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:34 smithi069 ceph-mon[53087]: failed to encode map e99 with expected crc 2024-03-26T02:01:35.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:34 smithi069 ceph-mon[53087]: failed to encode map e99 with expected crc 2024-03-26T02:01:35.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:34 smithi069 ceph-mon[53087]: failed to encode map e99 with expected crc 2024-03-26T02:01:35.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:34 smithi069 ceph-mon[53087]: failed to encode map e99 with expected crc 2024-03-26T02:01:35.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:34 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 114/1617 objects degraded (7.050%), 6 pgs degraded (PG_DEGRADED) 2024-03-26T02:01:35.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:34 smithi069 ceph-mon[53087]: osdmap e100: 8 total, 8 up, 8 in 2024-03-26T02:01:35.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:34 smithi069 ceph-mon[53087]: failed to encode map e100 with expected crc 2024-03-26T02:01:35.113 INFO:teuthology.orchestra.run.smithi019.stdout: 5 224300 10.92 MB/sec execute 368 sec latency 432.962 ms 2024-03-26T02:01:36.072 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:35 smithi019 ceph-mon[66359]: failed to encode map e100 with expected crc 2024-03-26T02:01:36.072 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:35 smithi019 ceph-mon[66359]: failed to encode map e100 with expected crc 2024-03-26T02:01:36.072 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:35 smithi019 ceph-mon[66359]: failed to encode map e100 with expected crc 2024-03-26T02:01:36.072 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:35 smithi019 ceph-mon[66359]: failed to encode map e100 with expected crc 2024-03-26T02:01:36.072 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:35 smithi019 ceph-mon[66359]: failed to encode map e100 with expected crc 2024-03-26T02:01:36.072 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:35 smithi019 ceph-mon[66359]: failed to encode map e100 with expected crc 2024-03-26T02:01:36.072 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:35 smithi019 ceph-mon[66359]: failed to encode map e100 with expected crc 2024-03-26T02:01:36.073 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:35 smithi019 ceph-mon[66359]: pgmap v122: 97 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 339 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.4 MiB/s wr, 401 op/s; 105/1632 objects degraded (6.434%); 1.6 MiB/s, 39 objects/s recovering 2024-03-26T02:01:36.073 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:35 smithi019 ceph-mon[66359]: osdmap e101: 8 total, 8 up, 8 in 2024-03-26T02:01:36.073 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:35 smithi019 ceph-mon[66359]: failed to encode map e101 with expected crc 2024-03-26T02:01:36.073 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:35 smithi019 ceph-mon[66359]: failed to encode map e101 with expected crc 2024-03-26T02:01:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:35 smithi069 ceph-mon[53087]: failed to encode map e100 with expected crc 2024-03-26T02:01:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:35 smithi069 ceph-mon[53087]: failed to encode map e100 with expected crc 2024-03-26T02:01:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:35 smithi069 ceph-mon[53087]: failed to encode map e100 with expected crc 2024-03-26T02:01:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:35 smithi069 ceph-mon[53087]: failed to encode map e100 with expected crc 2024-03-26T02:01:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:35 smithi069 ceph-mon[53087]: failed to encode map e100 with expected crc 2024-03-26T02:01:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:35 smithi069 ceph-mon[53087]: failed to encode map e100 with expected crc 2024-03-26T02:01:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:35 smithi069 ceph-mon[53087]: failed to encode map e100 with expected crc 2024-03-26T02:01:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:35 smithi069 ceph-mon[53087]: pgmap v122: 97 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 339 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.4 MiB/s wr, 401 op/s; 105/1632 objects degraded (6.434%); 1.6 MiB/s, 39 objects/s recovering 2024-03-26T02:01:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:35 smithi069 ceph-mon[53087]: osdmap e101: 8 total, 8 up, 8 in 2024-03-26T02:01:36.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:35 smithi069 ceph-mon[53087]: failed to encode map e101 with expected crc 2024-03-26T02:01:36.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:35 smithi069 ceph-mon[53087]: failed to encode map e101 with expected crc 2024-03-26T02:01:36.113 INFO:teuthology.orchestra.run.smithi019.stdout: 5 224569 10.91 MB/sec execute 369 sec latency 549.303 ms 2024-03-26T02:01:36.869 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:36 smithi019 ceph-mon[66359]: failed to encode map e101 with expected crc 2024-03-26T02:01:36.869 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:36 smithi019 ceph-mon[66359]: failed to encode map e101 with expected crc 2024-03-26T02:01:36.869 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:36 smithi019 ceph-mon[66359]: failed to encode map e101 with expected crc 2024-03-26T02:01:36.869 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:36 smithi019 ceph-mon[66359]: osdmap e102: 8 total, 8 up, 8 in 2024-03-26T02:01:36.869 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:36 smithi019 ceph-mon[66359]: failed to encode map e102 with expected crc 2024-03-26T02:01:37.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:36 smithi069 ceph-mon[53087]: failed to encode map e101 with expected crc 2024-03-26T02:01:37.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:36 smithi069 ceph-mon[53087]: failed to encode map e101 with expected crc 2024-03-26T02:01:37.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:36 smithi069 ceph-mon[53087]: failed to encode map e101 with expected crc 2024-03-26T02:01:37.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:36 smithi069 ceph-mon[53087]: osdmap e102: 8 total, 8 up, 8 in 2024-03-26T02:01:37.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:36 smithi069 ceph-mon[53087]: failed to encode map e102 with expected crc 2024-03-26T02:01:37.113 INFO:teuthology.orchestra.run.smithi019.stdout: 5 224804 10.89 MB/sec execute 370 sec latency 826.660 ms 2024-03-26T02:01:37.887 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:37 smithi019 ceph-mon[66359]: failed to encode map e102 with expected crc 2024-03-26T02:01:37.888 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:37 smithi019 ceph-mon[66359]: failed to encode map e102 with expected crc 2024-03-26T02:01:37.888 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:37 smithi019 ceph-mon[66359]: failed to encode map e102 with expected crc 2024-03-26T02:01:37.888 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:37 smithi019 ceph-mon[66359]: failed to encode map e102 with expected crc 2024-03-26T02:01:37.888 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:37 smithi019 ceph-mon[66359]: pgmap v125: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 342 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.4 MiB/s wr, 519 op/s; 9/1611 objects degraded (0.559%); 1024 KiB/s, 28 objects/s recovering 2024-03-26T02:01:37.888 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:37 smithi019 ceph-mon[66359]: osdmap e103: 8 total, 8 up, 8 in 2024-03-26T02:01:38.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:37 smithi069 ceph-mon[53087]: failed to encode map e102 with expected crc 2024-03-26T02:01:38.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:37 smithi069 ceph-mon[53087]: failed to encode map e102 with expected crc 2024-03-26T02:01:38.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:37 smithi069 ceph-mon[53087]: failed to encode map e102 with expected crc 2024-03-26T02:01:38.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:37 smithi069 ceph-mon[53087]: failed to encode map e102 with expected crc 2024-03-26T02:01:38.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:37 smithi069 ceph-mon[53087]: pgmap v125: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 342 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.4 MiB/s wr, 519 op/s; 9/1611 objects degraded (0.559%); 1024 KiB/s, 28 objects/s recovering 2024-03-26T02:01:38.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:37 smithi069 ceph-mon[53087]: osdmap e103: 8 total, 8 up, 8 in 2024-03-26T02:01:38.113 INFO:teuthology.orchestra.run.smithi019.stdout: 5 225082 10.88 MB/sec execute 371 sec latency 735.559 ms 2024-03-26T02:01:38.909 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:38 smithi019 ceph-mon[66359]: failed to encode map e103 with expected crc 2024-03-26T02:01:38.910 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:38 smithi019 ceph-mon[66359]: failed to encode map e103 with expected crc 2024-03-26T02:01:38.910 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:38 smithi019 ceph-mon[66359]: osdmap e104: 8 total, 8 up, 8 in 2024-03-26T02:01:39.114 INFO:teuthology.orchestra.run.smithi019.stdout: 5 225410 10.88 MB/sec execute 372 sec latency 1006.263 ms 2024-03-26T02:01:39.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:38 smithi069 ceph-mon[53087]: failed to encode map e103 with expected crc 2024-03-26T02:01:39.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:38 smithi069 ceph-mon[53087]: failed to encode map e103 with expected crc 2024-03-26T02:01:39.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:38 smithi069 ceph-mon[53087]: osdmap e104: 8 total, 8 up, 8 in 2024-03-26T02:01:39.932 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:39 smithi019 ceph-mon[66359]: failed to encode map e104 with expected crc 2024-03-26T02:01:39.932 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:39 smithi019 ceph-mon[66359]: failed to encode map e104 with expected crc 2024-03-26T02:01:39.932 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:39 smithi019 ceph-mon[66359]: failed to encode map e104 with expected crc 2024-03-26T02:01:39.932 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:39 smithi019 ceph-mon[66359]: failed to encode map e104 with expected crc 2024-03-26T02:01:39.932 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:39 smithi019 ceph-mon[66359]: pgmap v128: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 95 active+clean; 337 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.8 MiB/s wr, 556 op/s; 8/1608 objects degraded (0.498%); 0 B/s, 22 objects/s recovering 2024-03-26T02:01:39.932 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:39 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 9/1611 objects degraded (0.559%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T02:01:40.114 INFO:teuthology.orchestra.run.smithi019.stdout: 5 225804 10.89 MB/sec execute 373 sec latency 28.865 ms 2024-03-26T02:01:40.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:39 smithi069 ceph-mon[53087]: failed to encode map e104 with expected crc 2024-03-26T02:01:40.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:39 smithi069 ceph-mon[53087]: failed to encode map e104 with expected crc 2024-03-26T02:01:40.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:39 smithi069 ceph-mon[53087]: failed to encode map e104 with expected crc 2024-03-26T02:01:40.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:39 smithi069 ceph-mon[53087]: failed to encode map e104 with expected crc 2024-03-26T02:01:40.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:39 smithi069 ceph-mon[53087]: pgmap v128: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 95 active+clean; 337 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.8 MiB/s wr, 556 op/s; 8/1608 objects degraded (0.498%); 0 B/s, 22 objects/s recovering 2024-03-26T02:01:40.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:39 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 9/1611 objects degraded (0.559%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T02:01:41.114 INFO:teuthology.orchestra.run.smithi019.stdout: 5 226237 10.90 MB/sec execute 374 sec latency 31.967 ms 2024-03-26T02:01:41.973 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:41 smithi019 ceph-mon[66359]: pgmap v129: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 95 active+clean; 339 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.2 MiB/s wr, 491 op/s; 8/1605 objects degraded (0.498%); 0 B/s, 16 objects/s recovering 2024-03-26T02:01:42.114 INFO:teuthology.orchestra.run.smithi019.stdout: 5 226662 10.88 MB/sec execute 375 sec latency 41.592 ms 2024-03-26T02:01:42.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:41 smithi069 ceph-mon[53087]: pgmap v129: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 95 active+clean; 339 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.2 MiB/s wr, 491 op/s; 8/1605 objects degraded (0.498%); 0 B/s, 16 objects/s recovering 2024-03-26T02:01:42.981 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:42 smithi019 ceph-mon[66359]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8/1605 objects degraded (0.498%), 1 pg degraded) 2024-03-26T02:01:42.981 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:42 smithi019 ceph-mon[66359]: Cluster is now healthy 2024-03-26T02:01:43.114 INFO:teuthology.orchestra.run.smithi019.stdout: 5 227037 10.88 MB/sec execute 376 sec latency 45.494 ms 2024-03-26T02:01:43.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:42 smithi069 ceph-mon[53087]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8/1605 objects degraded (0.498%), 1 pg degraded) 2024-03-26T02:01:43.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:42 smithi069 ceph-mon[53087]: Cluster is now healthy 2024-03-26T02:01:43.988 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:43 smithi019 ceph-mon[66359]: pgmap v130: 97 pgs: 97 active+clean; 343 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.9 MiB/s wr, 634 op/s; 0 B/s, 16 objects/s recovering 2024-03-26T02:01:44.114 INFO:teuthology.orchestra.run.smithi019.stdout: 5 227485 10.88 MB/sec execute 377 sec latency 46.371 ms 2024-03-26T02:01:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:43 smithi069 ceph-mon[53087]: pgmap v130: 97 pgs: 97 active+clean; 343 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.9 MiB/s wr, 634 op/s; 0 B/s, 16 objects/s recovering 2024-03-26T02:01:45.114 INFO:teuthology.orchestra.run.smithi019.stdout: 5 227917 10.88 MB/sec execute 378 sec latency 35.387 ms 2024-03-26T02:01:46.034 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:45 smithi019 ceph-mon[66359]: pgmap v131: 97 pgs: 97 active+clean; 345 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 4.2 MiB/s wr, 393 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T02:01:46.034 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T02:01:46.034 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:46.034 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:01:46.114 INFO:teuthology.orchestra.run.smithi019.stdout: 5 228284 10.88 MB/sec execute 379 sec latency 59.151 ms 2024-03-26T02:01:46.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:45 smithi069 ceph-mon[53087]: pgmap v131: 97 pgs: 97 active+clean; 345 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 4.2 MiB/s wr, 393 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T02:01:46.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T02:01:46.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:46.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:01:47.055 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:46 smithi019 ceph-mon[66359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T02:01:47.055 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:46 smithi019 ceph-mon[66359]: Upgrade: osd.2 is safe to restart 2024-03-26T02:01:47.114 INFO:teuthology.orchestra.run.smithi019.stdout: 5 228665 10.88 MB/sec execute 380 sec latency 37.853 ms 2024-03-26T02:01:47.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:46 smithi069 ceph-mon[53087]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T02:01:47.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:46 smithi069 ceph-mon[53087]: Upgrade: osd.2 is safe to restart 2024-03-26T02:01:48.114 INFO:teuthology.orchestra.run.smithi019.stdout: 5 228962 10.88 MB/sec execute 381 sec latency 64.835 ms 2024-03-26T02:01:48.117 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:47 smithi019 ceph-mon[66359]: pgmap v132: 97 pgs: 97 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 2.9 KiB/s rd, 5.1 MiB/s wr, 499 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T02:01:48.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:47 smithi069 ceph-mon[53087]: pgmap v132: 97 pgs: 97 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 2.9 KiB/s rd, 5.1 MiB/s wr, 499 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T02:01:49.114 INFO:teuthology.orchestra.run.smithi019.stdout: 5 229239 10.88 MB/sec execute 382 sec latency 60.602 ms 2024-03-26T02:01:49.381 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:49 smithi019 ceph-mon[66359]: Upgrade: Updating osd.2 2024-03-26T02:01:49.381 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:49 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:49.381 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:49 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T02:01:49.381 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:49 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:01:49.381 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:49 smithi019 ceph-mon[66359]: Deploying daemon osd.2 on smithi019 2024-03-26T02:01:49.381 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:49 smithi019 ceph-mon[66359]: pgmap v133: 97 pgs: 97 active+clean; 337 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.1 MiB/s wr, 474 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T02:01:49.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:49 smithi069 ceph-mon[53087]: Upgrade: Updating osd.2 2024-03-26T02:01:49.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:49 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:49.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:49 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T02:01:49.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:49 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:01:49.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:49 smithi069 ceph-mon[53087]: Deploying daemon osd.2 on smithi019 2024-03-26T02:01:49.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:49 smithi069 ceph-mon[53087]: pgmap v133: 97 pgs: 97 active+clean; 337 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.1 MiB/s wr, 474 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T02:01:50.114 INFO:teuthology.orchestra.run.smithi019.stdout: 5 229414 10.87 MB/sec execute 383 sec latency 50.481 ms 2024-03-26T02:01:51.114 INFO:teuthology.orchestra.run.smithi019.stdout: 5 229667 10.86 MB/sec execute 384 sec latency 70.211 ms 2024-03-26T02:01:51.974 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:51 smithi019 ceph-mon[66359]: pgmap v134: 97 pgs: 97 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.8 MiB/s wr, 488 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T02:01:51.974 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:51 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:52.094 INFO:teuthology.orchestra.run.smithi019.stdout:true 2024-03-26T02:01:52.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:51 smithi069 ceph-mon[53087]: pgmap v134: 97 pgs: 97 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.8 MiB/s wr, 488 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T02:01:52.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:51 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:01:52.114 INFO:teuthology.orchestra.run.smithi019.stdout: 5 229919 10.84 MB/sec execute 385 sec latency 57.634 ms 2024-03-26T02:01:53.114 INFO:teuthology.orchestra.run.smithi019.stdout: 5 230192 10.82 MB/sec execute 386 sec latency 47.096 ms 2024-03-26T02:01:53.153 INFO:teuthology.orchestra.run.smithi019.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:01:53.154 INFO:teuthology.orchestra.run.smithi019.stdout:alertmanager.smithi019 smithi019 *:9093,9094 running (5m) 29s ago 15m 14.4M - 0.23.0 ba2b418f427c a6630841c2b2 2024-03-26T02:01:53.154 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi019 smithi019 running (2m) 29s ago 14m 6660k - 19.0.0-1775-g5e70124b 5f21c9264136 c5f91a47ddf2 2024-03-26T02:01:53.154 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi069 smithi069 running (119s) 115s ago 14m 6669k - 19.0.0-1775-g5e70124b 5f21c9264136 05128803b49e 2024-03-26T02:01:53.154 INFO:teuthology.orchestra.run.smithi019.stdout:grafana.smithi019 smithi019 *:3000 running (5m) 29s ago 14m 47.2M - 8.3.5 dad864ee21e9 a97dea3b5fec 2024-03-26T02:01:53.154 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi019.lmryom smithi019 running (9m) 29s ago 8m 38.3M - 17.2.0 e1d6a67b021e 91a7c2942715 2024-03-26T02:01:53.154 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi069.spgwgo smithi069 running (8m) 115s ago 8m 14.4M - 17.2.0 e1d6a67b021e 94d96338dbe1 2024-03-26T02:01:53.154 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi019.ixuvdm smithi019 *:8443,9283,8765 running (4m) 29s ago 16m 501M - 19.0.0-1775-g5e70124b 5f21c9264136 4460b22c7a56 2024-03-26T02:01:53.154 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi069.wpbewu smithi069 *:8443,9283,8765 running (3m) 115s ago 14m 451M - 19.0.0-1775-g5e70124b 5f21c9264136 380fd33842a8 2024-03-26T02:01:53.154 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi019 smithi019 running (3m) 29s ago 16m 49.5M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 c18934705774 2024-03-26T02:01:53.154 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi069 smithi069 running (3m) 115s ago 14m 35.6M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 a856bd263104 2024-03-26T02:01:53.154 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.0.0.smithi019.qbizxo smithi019 *:2049 running (8m) 29s ago 8m 283M - 4.0 e1d6a67b021e cd6628a9075d 2024-03-26T02:01:53.154 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.1.0.smithi069.uawdhc smithi069 *:2049 running (8m) 115s ago 8m 85.3M - 4.0 e1d6a67b021e 6ee3695e7a10 2024-03-26T02:01:53.154 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi019 smithi019 *:9100 running (5m) 29s ago 14m 9.76M - 1.3.1 1dbe0e931976 0d1d184bec63 2024-03-26T02:01:53.155 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi069 smithi069 *:9100 running (4m) 115s ago 14m 19.2M - 1.3.1 1dbe0e931976 c49a006f94e7 2024-03-26T02:01:53.155 INFO:teuthology.orchestra.run.smithi019.stdout:osd.0 smithi019 running (87s) 29s ago 13m 251M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 3f931cfe320a 2024-03-26T02:01:53.155 INFO:teuthology.orchestra.run.smithi019.stdout:osd.1 smithi019 running (39s) 29s ago 12m 15.2M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 758caf4ed841 2024-03-26T02:01:53.155 INFO:teuthology.orchestra.run.smithi019.stdout:osd.2 smithi019 running (12m) 29s ago 12m 521M 1937M 17.2.0 e1d6a67b021e f549115e7685 2024-03-26T02:01:53.155 INFO:teuthology.orchestra.run.smithi019.stdout:osd.3 smithi019 running (12m) 29s ago 12m 388M 1937M 17.2.0 e1d6a67b021e 154af68d03e1 2024-03-26T02:01:53.155 INFO:teuthology.orchestra.run.smithi019.stdout:osd.4 smithi069 running (11m) 115s ago 11m 556M 2704M 17.2.0 e1d6a67b021e 4fd5e853f8bf 2024-03-26T02:01:53.155 INFO:teuthology.orchestra.run.smithi019.stdout:osd.5 smithi069 running (11m) 115s ago 11m 433M 2704M 17.2.0 e1d6a67b021e 932e401d87c9 2024-03-26T02:01:53.155 INFO:teuthology.orchestra.run.smithi019.stdout:osd.6 smithi069 running (10m) 115s ago 10m 345M 2704M 17.2.0 e1d6a67b021e b123f75154c8 2024-03-26T02:01:53.155 INFO:teuthology.orchestra.run.smithi019.stdout:osd.7 smithi069 running (10m) 115s ago 10m 550M 2704M 17.2.0 e1d6a67b021e d175fbc9ff8a 2024-03-26T02:01:53.155 INFO:teuthology.orchestra.run.smithi019.stdout:prometheus.smithi019 smithi019 *:9095 running (3m) 29s ago 13m 60.2M - 2.33.4 514e6a882f6e 29f89db72a07 2024-03-26T02:01:53.762 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:01:53 smithi019 systemd[1]: Stopping Ceph osd.2 for 692607c6-eb12-11ee-95d0-87774f69a715... 2024-03-26T02:01:53.798 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:01:53.798 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": { 2024-03-26T02:01:53.799 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:01:53.799 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:01:53.799 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": { 2024-03-26T02:01:53.799 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:01:53.799 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:01:53.799 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": { 2024-03-26T02:01:53.799 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-03-26T02:01:53.799 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:01:53.799 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:01:53.799 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": { 2024-03-26T02:01:53.799 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:01:53.800 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:01:53.800 INFO:teuthology.orchestra.run.smithi019.stdout: "overall": { 2024-03-26T02:01:53.800 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-03-26T02:01:53.800 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 6 2024-03-26T02:01:53.800 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-26T02:01:53.800 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:01:54.013 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:53 smithi019 ceph-mon[66359]: from='client.44201 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:54.013 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:53 smithi019 ceph-mon[66359]: from='client.34270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:54.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:53 smithi019 ceph-mon[66359]: pgmap v135: 97 pgs: 97 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 6.3 MiB/s wr, 589 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T02:01:54.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:53 smithi069 ceph-mon[53087]: from='client.44201 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:54.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:53 smithi069 ceph-mon[53087]: from='client.34270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:54.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:53 smithi069 ceph-mon[53087]: pgmap v135: 97 pgs: 97 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 6.3 MiB/s wr, 589 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T02:01:54.117 INFO:teuthology.orchestra.run.smithi019.stdout: 5 230403 10.81 MB/sec execute 387 sec latency 68.755 ms 2024-03-26T02:01:54.279 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:01:54.279 INFO:teuthology.orchestra.run.smithi019.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2", 2024-03-26T02:01:54.280 INFO:teuthology.orchestra.run.smithi019.stdout: "in_progress": true, 2024-03-26T02:01:54.280 INFO:teuthology.orchestra.run.smithi019.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T02:01:54.280 INFO:teuthology.orchestra.run.smithi019.stdout: "services_complete": [ 2024-03-26T02:01:54.280 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr", 2024-03-26T02:01:54.280 INFO:teuthology.orchestra.run.smithi019.stdout: "crash", 2024-03-26T02:01:54.280 INFO:teuthology.orchestra.run.smithi019.stdout: "mon" 2024-03-26T02:01:54.280 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-26T02:01:54.280 INFO:teuthology.orchestra.run.smithi019.stdout: "progress": "8/23 daemons upgraded", 2024-03-26T02:01:54.280 INFO:teuthology.orchestra.run.smithi019.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T02:01:54.280 INFO:teuthology.orchestra.run.smithi019.stdout: "is_paused": false 2024-03-26T02:01:54.280 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:01:54.523 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:01:54 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2[43080]: 2024-03-26T02:01:54.455+0000 7fa1a4275700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:01:54.523 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:01:54 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2[43080]: 2024-03-26T02:01:54.455+0000 7fa1a4275700 -1 osd.2 104 *** Got signal Terminated *** 2024-03-26T02:01:54.523 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:01:54 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2[43080]: 2024-03-26T02:01:54.455+0000 7fa1a4275700 -1 osd.2 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:01:54.888 INFO:teuthology.orchestra.run.smithi019.stdout:HEALTH_WARN 1 osds down 2024-03-26T02:01:54.888 INFO:teuthology.orchestra.run.smithi019.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-26T02:01:54.888 INFO:teuthology.orchestra.run.smithi019.stdout: osd.2 (root=default,host=smithi019) is down 2024-03-26T02:01:55.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:54 smithi069 ceph-mon[53087]: from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:55.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:54 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/2048925568' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:01:55.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:54 smithi069 ceph-mon[53087]: osd.2 marked itself down and dead 2024-03-26T02:01:55.121 INFO:teuthology.orchestra.run.smithi019.stdout: 5 230593 10.79 MB/sec execute 388 sec latency 304.966 ms 2024-03-26T02:01:55.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:54 smithi019 ceph-mon[66359]: from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:55.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:54 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/2048925568' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:01:55.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:54 smithi019 ceph-mon[66359]: osd.2 marked itself down and dead 2024-03-26T02:01:55.735 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:01:55 smithi019 podman[77274]: 2024-03-26 02:01:55.460302156 +0000 UTC m=+1.575627956 container died f549115e7685e42cf99faf60d9603bd0ac2b53ea1220fa5bbd8741cd3698fc70 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, ceph=True, vendor=Red Hat, Inc., version=8, com.redhat.component=centos-stream-container, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, distribution-scope=public, io.buildah.version=1.19.8, architecture=x86_64, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, GIT_BRANCH=HEAD, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-03-26T02:01:55.992 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:01:55 smithi019 podman[77274]: 2024-03-26 02:01:55.851659287 +0000 UTC m=+1.966985095 container cleanup f549115e7685e42cf99faf60d9603bd0ac2b53ea1220fa5bbd8741cd3698fc70 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, ceph=True, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, version=8, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-03-26T02:01:55.993 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:01:55 smithi019 bash[77274]: ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2 2024-03-26T02:01:55.993 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:55 smithi019 ceph-mon[66359]: from='client.34282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:55.993 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:55 smithi019 ceph-mon[66359]: pgmap v136: 97 pgs: 97 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.5 MiB/s wr, 478 op/s 2024-03-26T02:01:55.993 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:55 smithi019 ceph-mon[66359]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:01:55.993 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:55 smithi019 ceph-mon[66359]: osdmap e105: 8 total, 7 up, 8 in 2024-03-26T02:01:55.993 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:55 smithi019 ceph-mon[66359]: failed to encode map e105 with expected crc 2024-03-26T02:01:55.993 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:55 smithi019 ceph-mon[66359]: failed to encode map e105 with expected crc 2024-03-26T02:01:55.993 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:55 smithi019 ceph-mon[66359]: failed to encode map e105 with expected crc 2024-03-26T02:01:55.993 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:55 smithi019 ceph-mon[66359]: failed to encode map e105 with expected crc 2024-03-26T02:01:55.993 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:55 smithi019 ceph-mon[66359]: failed to encode map e105 with expected crc 2024-03-26T02:01:55.994 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:55 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/903571957' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:01:56.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:55 smithi069 ceph-mon[53087]: from='client.34282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:56.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:55 smithi069 ceph-mon[53087]: pgmap v136: 97 pgs: 97 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.5 MiB/s wr, 478 op/s 2024-03-26T02:01:56.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:55 smithi069 ceph-mon[53087]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:01:56.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:55 smithi069 ceph-mon[53087]: osdmap e105: 8 total, 7 up, 8 in 2024-03-26T02:01:56.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:55 smithi069 ceph-mon[53087]: failed to encode map e105 with expected crc 2024-03-26T02:01:56.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:55 smithi069 ceph-mon[53087]: failed to encode map e105 with expected crc 2024-03-26T02:01:56.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:55 smithi069 ceph-mon[53087]: failed to encode map e105 with expected crc 2024-03-26T02:01:56.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:55 smithi069 ceph-mon[53087]: failed to encode map e105 with expected crc 2024-03-26T02:01:56.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:55 smithi069 ceph-mon[53087]: failed to encode map e105 with expected crc 2024-03-26T02:01:56.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:55 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/903571957' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:01:56.121 INFO:teuthology.orchestra.run.smithi019.stdout: 5 230690 10.77 MB/sec execute 389 sec latency 999.181 ms 2024-03-26T02:01:56.243 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:01:56 smithi019 podman[77337]: 2024-03-26 02:01:56.225788295 +0000 UTC m=+0.763082677 container remove f549115e7685e42cf99faf60d9603bd0ac2b53ea1220fa5bbd8741cd3698fc70 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, architecture=x86_64, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, name=centos-stream, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, io.k8s.display-name=CentOS Stream 8, release=754) 2024-03-26T02:01:57.011 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:01:56 smithi019 podman[77420]: 2024-03-26 02:01:56.708448384 +0000 UTC m=+0.028869953 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:01:57.012 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:01:56 smithi019 podman[77420]: 2024-03-26 02:01:56.881395084 +0000 UTC m=+0.201816684 container create cd884fd40450ffe2571ae09db7b252cf38e69241aa970bc986a1d2f28d5dd231 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, io.buildah.version=1.33.5) 2024-03-26T02:01:57.012 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:56 smithi019 ceph-mon[66359]: failed to encode map e106 with expected crc 2024-03-26T02:01:57.012 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:56 smithi019 ceph-mon[66359]: osdmap e106: 8 total, 7 up, 8 in 2024-03-26T02:01:57.012 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:56 smithi019 ceph-mon[66359]: failed to encode map e106 with expected crc 2024-03-26T02:01:57.012 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:56 smithi019 ceph-mon[66359]: failed to encode map e106 with expected crc 2024-03-26T02:01:57.012 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:56 smithi019 ceph-mon[66359]: failed to encode map e106 with expected crc 2024-03-26T02:01:57.012 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:56 smithi019 ceph-mon[66359]: failed to encode map e106 with expected crc 2024-03-26T02:01:57.012 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:56 smithi019 ceph-mon[66359]: failed to encode map e106 with expected crc 2024-03-26T02:01:57.012 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:56 smithi019 ceph-mon[66359]: failed to encode map e106 with expected crc 2024-03-26T02:01:57.013 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:56 smithi019 ceph-mon[66359]: failed to encode map e106 with expected crc 2024-03-26T02:01:57.013 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:56 smithi019 ceph-mon[66359]: failed to encode map e106 with expected crc 2024-03-26T02:01:57.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:56 smithi069 ceph-mon[53087]: failed to encode map e106 with expected crc 2024-03-26T02:01:57.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:56 smithi069 ceph-mon[53087]: osdmap e106: 8 total, 7 up, 8 in 2024-03-26T02:01:57.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:56 smithi069 ceph-mon[53087]: failed to encode map e106 with expected crc 2024-03-26T02:01:57.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:56 smithi069 ceph-mon[53087]: failed to encode map e106 with expected crc 2024-03-26T02:01:57.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:56 smithi069 ceph-mon[53087]: failed to encode map e106 with expected crc 2024-03-26T02:01:57.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:56 smithi069 ceph-mon[53087]: failed to encode map e106 with expected crc 2024-03-26T02:01:57.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:56 smithi069 ceph-mon[53087]: failed to encode map e106 with expected crc 2024-03-26T02:01:57.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:56 smithi069 ceph-mon[53087]: failed to encode map e106 with expected crc 2024-03-26T02:01:57.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:56 smithi069 ceph-mon[53087]: failed to encode map e106 with expected crc 2024-03-26T02:01:57.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:56 smithi069 ceph-mon[53087]: failed to encode map e106 with expected crc 2024-03-26T02:01:57.121 INFO:teuthology.orchestra.run.smithi019.stdout: 5 231012 10.76 MB/sec execute 390 sec latency 49.847 ms 2024-03-26T02:01:57.262 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:01:57 smithi019 podman[77420]: 2024-03-26 02:01:57.210558751 +0000 UTC m=+0.530980311 container init cd884fd40450ffe2571ae09db7b252cf38e69241aa970bc986a1d2f28d5dd231 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2-deactivate, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-26T02:01:57.263 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:01:57 smithi019 podman[77420]: 2024-03-26 02:01:57.217491939 +0000 UTC m=+0.537913493 container start cd884fd40450ffe2571ae09db7b252cf38e69241aa970bc986a1d2f28d5dd231 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124) 2024-03-26T02:01:57.513 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:01:57 smithi019 podman[77420]: 2024-03-26 02:01:57.410405469 +0000 UTC m=+0.730827020 container attach cd884fd40450ffe2571ae09db7b252cf38e69241aa970bc986a1d2f28d5dd231 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-26T02:01:57.765 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:01:57 smithi019 podman[77420]: 2024-03-26 02:01:57.666287202 +0000 UTC m=+0.986708755 container died cd884fd40450ffe2571ae09db7b252cf38e69241aa970bc986a1d2f28d5dd231 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-26T02:01:58.019 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:57 smithi019 ceph-mon[66359]: pgmap v139: 97 pgs: 3 active+undersized, 16 active+undersized+degraded, 2 stale+active+clean, 76 active+clean; 337 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.7 MiB/s wr, 527 op/s; 100/1542 objects degraded (6.485%) 2024-03-26T02:01:58.020 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:57 smithi019 ceph-mon[66359]: Health check failed: Degraded data redundancy: 100/1542 objects degraded (6.485%), 16 pgs degraded (PG_DEGRADED) 2024-03-26T02:01:58.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:57 smithi069 ceph-mon[53087]: pgmap v139: 97 pgs: 3 active+undersized, 16 active+undersized+degraded, 2 stale+active+clean, 76 active+clean; 337 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.7 MiB/s wr, 527 op/s; 100/1542 objects degraded (6.485%) 2024-03-26T02:01:58.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:57 smithi069 ceph-mon[53087]: Health check failed: Degraded data redundancy: 100/1542 objects degraded (6.485%), 16 pgs degraded (PG_DEGRADED) 2024-03-26T02:01:58.121 INFO:teuthology.orchestra.run.smithi019.stdout: 5 231349 10.75 MB/sec execute 391 sec latency 60.088 ms 2024-03-26T02:01:58.787 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:01:58 smithi019 podman[77443]: 2024-03-26 02:01:58.743375049 +0000 UTC m=+1.069947873 container remove cd884fd40450ffe2571ae09db7b252cf38e69241aa970bc986a1d2f28d5dd231 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319) 2024-03-26T02:01:58.787 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:01:58 smithi019 systemd[1]: ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.2.service: Deactivated successfully. 2024-03-26T02:01:58.787 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:01:58 smithi019 systemd[1]: Stopped Ceph osd.2 for 692607c6-eb12-11ee-95d0-87774f69a715. 2024-03-26T02:01:58.787 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:01:58 smithi019 systemd[1]: ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.2.service: Consumed 2min 51.816s CPU time. 2024-03-26T02:01:59.122 INFO:teuthology.orchestra.run.smithi019.stdout: 5 231682 10.75 MB/sec execute 392 sec latency 42.422 ms 2024-03-26T02:01:59.288 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:01:59 smithi019 systemd[1]: Starting Ceph osd.2 for 692607c6-eb12-11ee-95d0-87774f69a715... 2024-03-26T02:01:59.545 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:01:59 smithi019 podman[77537]: 2024-03-26 02:01:59.283321187 +0000 UTC m=+0.023831335 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:01:59.545 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:01:59 smithi019 podman[77537]: 2024-03-26 02:01:59.506432225 +0000 UTC m=+0.246942382 container create 2f62202a9d475ff54a62cb1c02a5513ac0c32edaa1b4953adf3727a8646a85c5 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2-activate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-26T02:02:00.050 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:01:59 smithi019 ceph-mon[66359]: pgmap v140: 97 pgs: 4 active+undersized, 17 active+undersized+degraded, 2 stale+active+clean, 74 active+clean; 335 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.4 MiB/s wr, 425 op/s; 113/1527 objects degraded (7.400%) 2024-03-26T02:02:00.050 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:01:59 smithi019 podman[77537]: 2024-03-26 02:01:59.874124231 +0000 UTC m=+0.614634386 container init 2f62202a9d475ff54a62cb1c02a5513ac0c32edaa1b4953adf3727a8646a85c5 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2-activate, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:02:00.050 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:01:59 smithi019 podman[77537]: 2024-03-26 02:01:59.883205312 +0000 UTC m=+0.623715467 container start 2f62202a9d475ff54a62cb1c02a5513ac0c32edaa1b4953adf3727a8646a85c5 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2-activate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-26T02:02:00.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:01:59 smithi069 ceph-mon[53087]: pgmap v140: 97 pgs: 4 active+undersized, 17 active+undersized+degraded, 2 stale+active+clean, 74 active+clean; 335 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.4 MiB/s wr, 425 op/s; 113/1527 objects degraded (7.400%) 2024-03-26T02:02:00.121 INFO:teuthology.orchestra.run.smithi019.stdout: 5 231969 10.74 MB/sec execute 393 sec latency 72.084 ms 2024-03-26T02:02:00.305 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:02:00 smithi019 podman[77537]: 2024-03-26 02:02:00.055190608 +0000 UTC m=+0.795700761 container attach 2f62202a9d475ff54a62cb1c02a5513ac0c32edaa1b4953adf3727a8646a85c5 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-26T02:02:00.569 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:02:00 smithi019 bash[77537]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-26T02:02:00.569 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:02:00 smithi019 bash[77537]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-26T02:02:00.569 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:02:00 smithi019 bash[77537]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-26T02:02:00.570 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:02:00 smithi019 bash[77537]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-26T02:02:00.570 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:02:00 smithi019 bash[77537]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-26T02:02:00.570 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:02:00 smithi019 bash[77537]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-26T02:02:00.570 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:02:00 smithi019 bash[77537]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-26T02:02:00.570 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:02:00 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2-activate[77550]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-26T02:02:00.570 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:02:00 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2-activate[77550]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-26T02:02:00.570 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:02:00 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2-activate[77550]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-26T02:02:00.570 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:02:00 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2-activate[77550]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-26T02:02:00.570 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:02:00 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2-activate[77550]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-26T02:02:00.570 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:02:00 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2-activate[77550]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-26T02:02:00.570 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:02:00 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2-activate[77550]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-26T02:02:00.570 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:02:00 smithi019 podman[77537]: 2024-03-26 02:02:00.537755098 +0000 UTC m=+1.278265257 container died 2f62202a9d475ff54a62cb1c02a5513ac0c32edaa1b4953adf3727a8646a85c5 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2-activate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-26T02:02:01.122 INFO:teuthology.orchestra.run.smithi019.stdout: 5 232454 10.76 MB/sec execute 394 sec latency 35.641 ms 2024-03-26T02:02:01.836 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:01 smithi019 ceph-mon[66359]: pgmap v141: 97 pgs: 6 active+undersized, 20 active+undersized+degraded, 71 active+clean; 335 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.9 MiB/s wr, 233 op/s; 144/1533 objects degraded (9.393%) 2024-03-26T02:02:01.837 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:01.837 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:02:01.837 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:02:01 smithi019 podman[77682]: 2024-03-26 02:02:01.624001526 +0000 UTC m=+1.077838930 container remove 2f62202a9d475ff54a62cb1c02a5513ac0c32edaa1b4953adf3727a8646a85c5 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-26T02:02:02.088 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:02:01 smithi019 podman[77740]: 2024-03-26 02:02:01.785945739 +0000 UTC m=+0.026743660 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:02:02.088 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:02:02 smithi019 podman[77740]: 2024-03-26 02:02:02.014849259 +0000 UTC m=+0.255647228 container create 6a9b701517ca8a8d5a0da2a333a2594a2dc289baf7918e273b577e3191576471 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5) 2024-03-26T02:02:02.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:01 smithi069 ceph-mon[53087]: pgmap v141: 97 pgs: 6 active+undersized, 20 active+undersized+degraded, 71 active+clean; 335 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.9 MiB/s wr, 233 op/s; 144/1533 objects degraded (9.393%) 2024-03-26T02:02:02.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:02.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:02:02.121 INFO:teuthology.orchestra.run.smithi019.stdout: 5 232869 10.76 MB/sec execute 395 sec latency 31.821 ms 2024-03-26T02:02:02.848 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:02 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 149/1692 objects degraded (8.806%), 20 pgs degraded (PG_DEGRADED) 2024-03-26T02:02:03.103 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:02:02 smithi019 podman[77740]: 2024-03-26 02:02:02.881193295 +0000 UTC m=+1.121991215 container init 6a9b701517ca8a8d5a0da2a333a2594a2dc289baf7918e273b577e3191576471 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-26T02:02:03.104 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:02:02 smithi019 podman[77740]: 2024-03-26 02:02:02.888030951 +0000 UTC m=+1.128828862 container start 6a9b701517ca8a8d5a0da2a333a2594a2dc289baf7918e273b577e3191576471 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-26T02:02:03.104 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:02:02 smithi019 ceph-osd[77759]: -- 172.21.15.19:0/1211798864 <== mon.1 v2:172.21.15.69:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bbe5e4d1e0 con 0x55bbe5e1f000 2024-03-26T02:02:03.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:02 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 149/1692 objects degraded (8.806%), 20 pgs degraded (PG_DEGRADED) 2024-03-26T02:02:03.121 INFO:teuthology.orchestra.run.smithi019.stdout: 5 233249 10.77 MB/sec execute 396 sec latency 43.677 ms 2024-03-26T02:02:03.357 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:02:03 smithi019 bash[77740]: 6a9b701517ca8a8d5a0da2a333a2594a2dc289baf7918e273b577e3191576471 2024-03-26T02:02:03.358 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:02:03 smithi019 systemd[1]: Started Ceph osd.2 for 692607c6-eb12-11ee-95d0-87774f69a715. 2024-03-26T02:02:03.557 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:02:03 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2[77755]: 2024-03-26T02:02:03.457+0000 7f610b3e07c0 -1 Falling back to public interface 2024-03-26T02:02:03.882 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:03 smithi019 ceph-mon[66359]: pgmap v142: 97 pgs: 6 active+undersized, 20 active+undersized+degraded, 71 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.9 MiB/s wr, 452 op/s; 149/1692 objects degraded (8.806%) 2024-03-26T02:02:03.883 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:03 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:03.883 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:03 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:03.883 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:03 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:02:04.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:03 smithi069 ceph-mon[53087]: pgmap v142: 97 pgs: 6 active+undersized, 20 active+undersized+degraded, 71 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.9 MiB/s wr, 452 op/s; 149/1692 objects degraded (8.806%) 2024-03-26T02:02:04.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:03 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:04.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:03 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:04.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:03 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:02:04.121 INFO:teuthology.orchestra.run.smithi019.stdout: 5 233515 10.75 MB/sec execute 397 sec latency 64.992 ms 2024-03-26T02:02:05.122 INFO:teuthology.orchestra.run.smithi019.stdout: 5 233822 10.74 MB/sec execute 398 sec latency 48.318 ms 2024-03-26T02:02:05.927 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:05 smithi019 ceph-mon[66359]: pgmap v143: 97 pgs: 6 active+undersized, 20 active+undersized+degraded, 71 active+clean; 343 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.8 MiB/s wr, 539 op/s; 152/1695 objects degraded (8.968%) 2024-03-26T02:02:06.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:05 smithi069 ceph-mon[53087]: pgmap v143: 97 pgs: 6 active+undersized, 20 active+undersized+degraded, 71 active+clean; 343 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.8 MiB/s wr, 539 op/s; 152/1695 objects degraded (8.968%) 2024-03-26T02:02:06.121 INFO:teuthology.orchestra.run.smithi019.stdout: 5 234096 10.73 MB/sec execute 399 sec latency 46.220 ms 2024-03-26T02:02:07.122 INFO:teuthology.orchestra.run.smithi019.stdout: 5 234465 10.73 MB/sec execute 400 sec latency 57.781 ms 2024-03-26T02:02:07.980 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:07 smithi019 ceph-mon[66359]: pgmap v144: 97 pgs: 6 active+undersized, 20 active+undersized+degraded, 71 active+clean; 341 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.5 MiB/s wr, 569 op/s; 149/1644 objects degraded (9.063%) 2024-03-26T02:02:08.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:07 smithi069 ceph-mon[53087]: pgmap v144: 97 pgs: 6 active+undersized, 20 active+undersized+degraded, 71 active+clean; 341 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.5 MiB/s wr, 569 op/s; 149/1644 objects degraded (9.063%) 2024-03-26T02:02:08.121 INFO:teuthology.orchestra.run.smithi019.stdout: 5 234862 10.72 MB/sec execute 401 sec latency 57.837 ms 2024-03-26T02:02:08.996 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:08 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 153/1632 objects degraded (9.375%), 20 pgs degraded (PG_DEGRADED) 2024-03-26T02:02:09.122 INFO:teuthology.orchestra.run.smithi019.stdout: 5 235237 10.72 MB/sec execute 402 sec latency 43.228 ms 2024-03-26T02:02:09.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:08 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 153/1632 objects degraded (9.375%), 20 pgs degraded (PG_DEGRADED) 2024-03-26T02:02:09.512 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:02:09 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2[77755]: 2024-03-26T02:02:09.431+0000 7f610b3e07c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-03-26T02:02:10.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:09 smithi019 ceph-mon[66359]: pgmap v145: 97 pgs: 6 active+undersized, 20 active+undersized+degraded, 71 active+clean; 341 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.3 MiB/s wr, 400 op/s; 153/1632 objects degraded (9.375%) 2024-03-26T02:02:10.122 INFO:teuthology.orchestra.run.smithi019.stdout: 5 235540 10.71 MB/sec execute 403 sec latency 50.210 ms 2024-03-26T02:02:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:09 smithi069 ceph-mon[53087]: pgmap v145: 97 pgs: 6 active+undersized, 20 active+undersized+degraded, 71 active+clean; 341 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.3 MiB/s wr, 400 op/s; 153/1632 objects degraded (9.375%) 2024-03-26T02:02:10.665 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:02:10 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2[77755]: 2024-03-26T02:02:10.631+0000 7f610b3e07c0 -1 osd.2 104 log_to_monitors true 2024-03-26T02:02:10.916 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:10 smithi019 ceph-mon[66359]: from='osd.2 [v2:172.21.15.19:6818/314629709,v1:172.21.15.19:6819/314629709]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T02:02:11.122 INFO:teuthology.orchestra.run.smithi019.stdout: 5 235904 10.72 MB/sec execute 404 sec latency 118.354 ms 2024-03-26T02:02:11.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:10 smithi069 ceph-mon[53087]: from='osd.2 [v2:172.21.15.19:6818/314629709,v1:172.21.15.19:6819/314629709]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T02:02:11.424 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:02:11 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2[77755]: 2024-03-26T02:02:11.416+0000 7f60f8830700 -1 osd.2 104 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:02:11.934 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:11 smithi019 ceph-mon[66359]: pgmap v146: 97 pgs: 6 active+undersized, 20 active+undersized+degraded, 71 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.4 MiB/s wr, 429 op/s; 156/1653 objects degraded (9.437%) 2024-03-26T02:02:11.934 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:11 smithi019 ceph-mon[66359]: from='osd.2 [v2:172.21.15.19:6818/314629709,v1:172.21.15.19:6819/314629709]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T02:02:11.934 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:11 smithi019 ceph-mon[66359]: osdmap e107: 8 total, 7 up, 8 in 2024-03-26T02:02:11.934 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:11 smithi019 ceph-mon[66359]: from='osd.2 [v2:172.21.15.19:6818/314629709,v1:172.21.15.19:6819/314629709]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-03-26T02:02:11.934 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:11 smithi019 ceph-mon[66359]: failed to encode map e107 with expected crc 2024-03-26T02:02:11.934 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:11 smithi019 ceph-mon[66359]: failed to encode map e107 with expected crc 2024-03-26T02:02:11.934 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:11 smithi019 ceph-mon[66359]: from='osd.2 [v2:172.21.15.19:6818/314629709,v1:172.21.15.19:6819/314629709]' entity='osd.2' 2024-03-26T02:02:12.122 INFO:teuthology.orchestra.run.smithi019.stdout: 5 236234 10.72 MB/sec execute 405 sec latency 118.943 ms 2024-03-26T02:02:12.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:11 smithi069 ceph-mon[53087]: pgmap v146: 97 pgs: 6 active+undersized, 20 active+undersized+degraded, 71 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.4 MiB/s wr, 429 op/s; 156/1653 objects degraded (9.437%) 2024-03-26T02:02:12.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:11 smithi069 ceph-mon[53087]: from='osd.2 [v2:172.21.15.19:6818/314629709,v1:172.21.15.19:6819/314629709]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T02:02:12.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:11 smithi069 ceph-mon[53087]: osdmap e107: 8 total, 7 up, 8 in 2024-03-26T02:02:12.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:11 smithi069 ceph-mon[53087]: from='osd.2 [v2:172.21.15.19:6818/314629709,v1:172.21.15.19:6819/314629709]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-03-26T02:02:12.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:11 smithi069 ceph-mon[53087]: failed to encode map e107 with expected crc 2024-03-26T02:02:12.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:11 smithi069 ceph-mon[53087]: failed to encode map e107 with expected crc 2024-03-26T02:02:12.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:11 smithi069 ceph-mon[53087]: from='osd.2 [v2:172.21.15.19:6818/314629709,v1:172.21.15.19:6819/314629709]' entity='osd.2' 2024-03-26T02:02:13.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:12 smithi019 ceph-mon[66359]: failed to encode map e107 with expected crc 2024-03-26T02:02:13.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:12 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:13.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:12 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:13.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:12 smithi019 ceph-mon[66359]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:02:13.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:12 smithi019 ceph-mon[66359]: osd.2 [v2:172.21.15.19:6818/314629709,v1:172.21.15.19:6819/314629709] boot 2024-03-26T02:02:13.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:12 smithi019 ceph-mon[66359]: osdmap e108: 8 total, 8 up, 8 in 2024-03-26T02:02:13.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:12 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:02:13.122 INFO:teuthology.orchestra.run.smithi019.stdout: 5 236513 10.72 MB/sec execute 406 sec latency 620.291 ms 2024-03-26T02:02:13.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:12 smithi069 ceph-mon[53087]: failed to encode map e107 with expected crc 2024-03-26T02:02:13.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:12 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:13.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:12 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:13.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:12 smithi069 ceph-mon[53087]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:02:13.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:12 smithi069 ceph-mon[53087]: osd.2 [v2:172.21.15.19:6818/314629709,v1:172.21.15.19:6819/314629709] boot 2024-03-26T02:02:13.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:12 smithi069 ceph-mon[53087]: osdmap e108: 8 total, 8 up, 8 in 2024-03-26T02:02:13.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:12 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:02:14.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:13 smithi019 ceph-mon[66359]: failed to encode map e108 with expected crc 2024-03-26T02:02:14.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:13 smithi019 ceph-mon[66359]: pgmap v149: 97 pgs: 6 active+undersized, 20 active+undersized+degraded, 71 active+clean; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 5.5 MiB/s wr, 490 op/s; 156/1671 objects degraded (9.336%) 2024-03-26T02:02:14.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:13 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:14.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:13 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:14.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:13 smithi019 ceph-mon[66359]: osdmap e109: 8 total, 8 up, 8 in 2024-03-26T02:02:14.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:13 smithi019 ceph-mon[66359]: failed to encode map e109 with expected crc 2024-03-26T02:02:14.017 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:13 smithi019 ceph-mon[66359]: failed to encode map e109 with expected crc 2024-03-26T02:02:14.017 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:13 smithi019 ceph-mon[66359]: failed to encode map e109 with expected crc 2024-03-26T02:02:14.122 INFO:teuthology.orchestra.run.smithi019.stdout: 5 236590 10.69 MB/sec execute 407 sec latency 949.863 ms 2024-03-26T02:02:14.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:13 smithi069 ceph-mon[53087]: failed to encode map e108 with expected crc 2024-03-26T02:02:14.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:13 smithi069 ceph-mon[53087]: pgmap v149: 97 pgs: 6 active+undersized, 20 active+undersized+degraded, 71 active+clean; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 5.5 MiB/s wr, 490 op/s; 156/1671 objects degraded (9.336%) 2024-03-26T02:02:14.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:13 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:14.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:13 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:14.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:13 smithi069 ceph-mon[53087]: osdmap e109: 8 total, 8 up, 8 in 2024-03-26T02:02:14.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:13 smithi069 ceph-mon[53087]: failed to encode map e109 with expected crc 2024-03-26T02:02:14.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:13 smithi069 ceph-mon[53087]: failed to encode map e109 with expected crc 2024-03-26T02:02:14.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:13 smithi069 ceph-mon[53087]: failed to encode map e109 with expected crc 2024-03-26T02:02:14.934 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:14 smithi019 ceph-mon[66359]: failed to encode map e109 with expected crc 2024-03-26T02:02:14.934 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:14 smithi019 ceph-mon[66359]: failed to encode map e109 with expected crc 2024-03-26T02:02:14.934 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:14 smithi019 ceph-mon[66359]: failed to encode map e109 with expected crc 2024-03-26T02:02:14.934 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:14 smithi019 ceph-mon[66359]: failed to encode map e109 with expected crc 2024-03-26T02:02:14.934 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:14 smithi019 ceph-mon[66359]: failed to encode map e109 with expected crc 2024-03-26T02:02:14.934 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:14 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 156/1671 objects degraded (9.336%), 20 pgs degraded (PG_DEGRADED) 2024-03-26T02:02:14.934 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:14 smithi019 ceph-mon[66359]: failed to encode map e110 with expected crc 2024-03-26T02:02:14.934 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:14 smithi019 ceph-mon[66359]: osdmap e110: 8 total, 8 up, 8 in 2024-03-26T02:02:14.934 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:14 smithi019 ceph-mon[66359]: failed to encode map e110 with expected crc 2024-03-26T02:02:15.122 INFO:teuthology.orchestra.run.smithi019.stdout: 5 236743 10.68 MB/sec execute 408 sec latency 1150.308 ms 2024-03-26T02:02:15.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:14 smithi069 ceph-mon[53087]: failed to encode map e109 with expected crc 2024-03-26T02:02:15.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:14 smithi069 ceph-mon[53087]: failed to encode map e109 with expected crc 2024-03-26T02:02:15.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:14 smithi069 ceph-mon[53087]: failed to encode map e109 with expected crc 2024-03-26T02:02:15.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:14 smithi069 ceph-mon[53087]: failed to encode map e109 with expected crc 2024-03-26T02:02:15.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:14 smithi069 ceph-mon[53087]: failed to encode map e109 with expected crc 2024-03-26T02:02:15.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:14 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 156/1671 objects degraded (9.336%), 20 pgs degraded (PG_DEGRADED) 2024-03-26T02:02:15.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:14 smithi069 ceph-mon[53087]: failed to encode map e110 with expected crc 2024-03-26T02:02:15.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:14 smithi069 ceph-mon[53087]: osdmap e110: 8 total, 8 up, 8 in 2024-03-26T02:02:15.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:14 smithi069 ceph-mon[53087]: failed to encode map e110 with expected crc 2024-03-26T02:02:15.955 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:15 smithi019 ceph-mon[66359]: failed to encode map e110 with expected crc 2024-03-26T02:02:15.955 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:15 smithi019 ceph-mon[66359]: failed to encode map e110 with expected crc 2024-03-26T02:02:15.955 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:15 smithi019 ceph-mon[66359]: failed to encode map e110 with expected crc 2024-03-26T02:02:15.955 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:15 smithi019 ceph-mon[66359]: pgmap v152: 97 pgs: 5 active+undersized, 2 active+recovery_wait+undersized+degraded+remapped, 18 active+undersized+degraded, 72 active+clean; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 6.3 MiB/s wr, 607 op/s; 175/1683 objects degraded (10.398%) 2024-03-26T02:02:15.956 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:15 smithi019 ceph-mon[66359]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T02:02:15.956 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:15 smithi019 ceph-mon[66359]: failed to encode map e111 with expected crc 2024-03-26T02:02:15.956 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:15 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:15.956 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:15 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:02:16.122 INFO:teuthology.orchestra.run.smithi019.stdout: 5 237124 10.68 MB/sec execute 409 sec latency 35.242 ms 2024-03-26T02:02:16.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:15 smithi069 ceph-mon[53087]: failed to encode map e110 with expected crc 2024-03-26T02:02:16.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:15 smithi069 ceph-mon[53087]: failed to encode map e110 with expected crc 2024-03-26T02:02:16.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:15 smithi069 ceph-mon[53087]: failed to encode map e110 with expected crc 2024-03-26T02:02:16.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:15 smithi069 ceph-mon[53087]: pgmap v152: 97 pgs: 5 active+undersized, 2 active+recovery_wait+undersized+degraded+remapped, 18 active+undersized+degraded, 72 active+clean; 346 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 6.3 MiB/s wr, 607 op/s; 175/1683 objects degraded (10.398%) 2024-03-26T02:02:16.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:15 smithi069 ceph-mon[53087]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T02:02:16.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:15 smithi069 ceph-mon[53087]: failed to encode map e111 with expected crc 2024-03-26T02:02:16.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:15 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:16.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:15 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:02:16.970 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:16 smithi019 ceph-mon[66359]: failed to encode map e111 with expected crc 2024-03-26T02:02:16.970 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:16 smithi019 ceph-mon[66359]: failed to encode map e111 with expected crc 2024-03-26T02:02:16.970 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:16 smithi019 ceph-mon[66359]: failed to encode map e111 with expected crc 2024-03-26T02:02:16.970 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:16 smithi019 ceph-mon[66359]: failed to encode map e111 with expected crc 2024-03-26T02:02:16.970 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:16 smithi019 ceph-mon[66359]: osdmap e112: 8 total, 8 up, 8 in 2024-03-26T02:02:17.122 INFO:teuthology.orchestra.run.smithi019.stdout: 5 237455 10.67 MB/sec execute 410 sec latency 713.128 ms 2024-03-26T02:02:17.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:16 smithi069 ceph-mon[53087]: failed to encode map e111 with expected crc 2024-03-26T02:02:17.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:16 smithi069 ceph-mon[53087]: failed to encode map e111 with expected crc 2024-03-26T02:02:17.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:16 smithi069 ceph-mon[53087]: failed to encode map e111 with expected crc 2024-03-26T02:02:17.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:16 smithi069 ceph-mon[53087]: failed to encode map e111 with expected crc 2024-03-26T02:02:17.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:16 smithi069 ceph-mon[53087]: osdmap e112: 8 total, 8 up, 8 in 2024-03-26T02:02:17.976 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:17 smithi019 ceph-mon[66359]: failed to encode map e112 with expected crc 2024-03-26T02:02:17.976 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:17 smithi019 ceph-mon[66359]: failed to encode map e112 with expected crc 2024-03-26T02:02:17.976 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:17 smithi019 ceph-mon[66359]: failed to encode map e112 with expected crc 2024-03-26T02:02:17.976 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:17 smithi019 ceph-mon[66359]: pgmap v155: 97 pgs: 1 active+recovery_wait+degraded, 2 peering, 1 active+undersized, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded, 88 active+clean; 342 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 4.0 MiB/s wr, 436 op/s; 76/1659 objects degraded (4.581%); 2.0 MiB/s, 18 objects/s recovering 2024-03-26T02:02:17.976 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:17 smithi019 ceph-mon[66359]: osdmap e113: 8 total, 8 up, 8 in 2024-03-26T02:02:17.976 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:17 smithi019 ceph-mon[66359]: failed to encode map e113 with expected crc 2024-03-26T02:02:18.122 INFO:teuthology.orchestra.run.smithi019.stdout: 5 237841 10.66 MB/sec execute 411 sec latency 1017.885 ms 2024-03-26T02:02:18.362 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:17 smithi069 ceph-mon[53087]: failed to encode map e112 with expected crc 2024-03-26T02:02:18.362 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:17 smithi069 ceph-mon[53087]: failed to encode map e112 with expected crc 2024-03-26T02:02:18.362 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:17 smithi069 ceph-mon[53087]: failed to encode map e112 with expected crc 2024-03-26T02:02:18.362 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:17 smithi069 ceph-mon[53087]: pgmap v155: 97 pgs: 1 active+recovery_wait+degraded, 2 peering, 1 active+undersized, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded, 88 active+clean; 342 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 4.0 MiB/s wr, 436 op/s; 76/1659 objects degraded (4.581%); 2.0 MiB/s, 18 objects/s recovering 2024-03-26T02:02:18.362 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:17 smithi069 ceph-mon[53087]: osdmap e113: 8 total, 8 up, 8 in 2024-03-26T02:02:18.362 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:17 smithi069 ceph-mon[53087]: failed to encode map e113 with expected crc 2024-03-26T02:02:19.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:18 smithi019 ceph-mon[66359]: failed to encode map e113 with expected crc 2024-03-26T02:02:19.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:18 smithi019 ceph-mon[66359]: failed to encode map e113 with expected crc 2024-03-26T02:02:19.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:18 smithi019 ceph-mon[66359]: failed to encode map e113 with expected crc 2024-03-26T02:02:19.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:18 smithi019 ceph-mon[66359]: failed to encode map e113 with expected crc 2024-03-26T02:02:19.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:18 smithi019 ceph-mon[66359]: osdmap e114: 8 total, 8 up, 8 in 2024-03-26T02:02:19.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:18 smithi019 ceph-mon[66359]: failed to encode map e114 with expected crc 2024-03-26T02:02:19.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:18 smithi019 ceph-mon[66359]: failed to encode map e114 with expected crc 2024-03-26T02:02:19.017 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:18 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:19.017 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:18 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:19.017 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:18 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:02:19.017 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:18 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:02:19.122 INFO:teuthology.orchestra.run.smithi019.stdout: 5 238149 10.66 MB/sec execute 412 sec latency 669.938 ms 2024-03-26T02:02:19.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:18 smithi069 ceph-mon[53087]: failed to encode map e113 with expected crc 2024-03-26T02:02:19.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:18 smithi069 ceph-mon[53087]: failed to encode map e113 with expected crc 2024-03-26T02:02:19.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:18 smithi069 ceph-mon[53087]: failed to encode map e113 with expected crc 2024-03-26T02:02:19.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:18 smithi069 ceph-mon[53087]: failed to encode map e113 with expected crc 2024-03-26T02:02:19.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:18 smithi069 ceph-mon[53087]: osdmap e114: 8 total, 8 up, 8 in 2024-03-26T02:02:19.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:18 smithi069 ceph-mon[53087]: failed to encode map e114 with expected crc 2024-03-26T02:02:19.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:18 smithi069 ceph-mon[53087]: failed to encode map e114 with expected crc 2024-03-26T02:02:19.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:18 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:19.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:18 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:19.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:18 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:02:19.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:18 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:02:20.044 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:19 smithi019 ceph-mon[66359]: failed to encode map e114 with expected crc 2024-03-26T02:02:20.044 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:19 smithi019 ceph-mon[66359]: failed to encode map e114 with expected crc 2024-03-26T02:02:20.044 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:19 smithi019 ceph-mon[66359]: failed to encode map e114 with expected crc 2024-03-26T02:02:20.045 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:19 smithi019 ceph-mon[66359]: failed to encode map e114 with expected crc 2024-03-26T02:02:20.045 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:19 smithi019 ceph-mon[66359]: failed to encode map e114 with expected crc 2024-03-26T02:02:20.045 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:19 smithi019 ceph-mon[66359]: failed to encode map e114 with expected crc 2024-03-26T02:02:20.045 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:19 smithi019 ceph-mon[66359]: failed to encode map e114 with expected crc 2024-03-26T02:02:20.045 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:19 smithi019 ceph-mon[66359]: failed to encode map e114 with expected crc 2024-03-26T02:02:20.045 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:19 smithi019 ceph-mon[66359]: failed to encode map e114 with expected crc 2024-03-26T02:02:20.045 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:19 smithi019 ceph-mon[66359]: pgmap v158: 97 pgs: 1 active+recovery_wait+degraded, 2 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 342 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 4.6 MiB/s wr, 455 op/s; 61/1656 objects degraded (3.684%); 2.2 MiB/s, 25 objects/s recovering 2024-03-26T02:02:20.045 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:19 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:20.046 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:19 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:20.046 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:19 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:02:20.046 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:19 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:02:20.046 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:19 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:02:20.046 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:19 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:02:20.046 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:19 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T02:02:20.046 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:19 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 76/1659 objects degraded (4.581%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T02:02:20.046 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:19 smithi019 ceph-mon[66359]: osdmap e115: 8 total, 8 up, 8 in 2024-03-26T02:02:20.122 INFO:teuthology.orchestra.run.smithi019.stdout: 5 238455 10.65 MB/sec execute 413 sec latency 854.287 ms 2024-03-26T02:02:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:19 smithi069 ceph-mon[53087]: failed to encode map e114 with expected crc 2024-03-26T02:02:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:19 smithi069 ceph-mon[53087]: failed to encode map e114 with expected crc 2024-03-26T02:02:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:19 smithi069 ceph-mon[53087]: failed to encode map e114 with expected crc 2024-03-26T02:02:20.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:19 smithi069 ceph-mon[53087]: failed to encode map e114 with expected crc 2024-03-26T02:02:20.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:19 smithi069 ceph-mon[53087]: failed to encode map e114 with expected crc 2024-03-26T02:02:20.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:19 smithi069 ceph-mon[53087]: failed to encode map e114 with expected crc 2024-03-26T02:02:20.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:19 smithi069 ceph-mon[53087]: failed to encode map e114 with expected crc 2024-03-26T02:02:20.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:19 smithi069 ceph-mon[53087]: failed to encode map e114 with expected crc 2024-03-26T02:02:20.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:19 smithi069 ceph-mon[53087]: failed to encode map e114 with expected crc 2024-03-26T02:02:20.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:19 smithi069 ceph-mon[53087]: pgmap v158: 97 pgs: 1 active+recovery_wait+degraded, 2 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 342 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 4.6 MiB/s wr, 455 op/s; 61/1656 objects degraded (3.684%); 2.2 MiB/s, 25 objects/s recovering 2024-03-26T02:02:20.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:19 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:20.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:19 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:20.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:19 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:02:20.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:19 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:02:20.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:19 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:02:20.361 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:19 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:02:20.361 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:19 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T02:02:20.361 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:19 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 76/1659 objects degraded (4.581%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T02:02:20.361 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:19 smithi069 ceph-mon[53087]: osdmap e115: 8 total, 8 up, 8 in 2024-03-26T02:02:21.049 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:20 smithi019 ceph-mon[66359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T02:02:21.049 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:20 smithi019 ceph-mon[66359]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-03-26T02:02:21.049 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:20 smithi019 ceph-mon[66359]: failed to encode map e115 with expected crc 2024-03-26T02:02:21.049 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:20 smithi019 ceph-mon[66359]: failed to encode map e115 with expected crc 2024-03-26T02:02:21.049 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:20 smithi019 ceph-mon[66359]: failed to encode map e115 with expected crc 2024-03-26T02:02:21.049 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:20 smithi019 ceph-mon[66359]: failed to encode map e115 with expected crc 2024-03-26T02:02:21.049 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:20 smithi019 ceph-mon[66359]: failed to encode map e115 with expected crc 2024-03-26T02:02:21.122 INFO:teuthology.orchestra.run.smithi019.stdout: 5 238843 10.65 MB/sec execute 414 sec latency 120.413 ms 2024-03-26T02:02:21.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:20 smithi069 ceph-mon[53087]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T02:02:21.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:20 smithi069 ceph-mon[53087]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-03-26T02:02:21.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:20 smithi069 ceph-mon[53087]: failed to encode map e115 with expected crc 2024-03-26T02:02:21.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:20 smithi069 ceph-mon[53087]: failed to encode map e115 with expected crc 2024-03-26T02:02:21.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:20 smithi069 ceph-mon[53087]: failed to encode map e115 with expected crc 2024-03-26T02:02:21.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:20 smithi069 ceph-mon[53087]: failed to encode map e115 with expected crc 2024-03-26T02:02:21.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:20 smithi069 ceph-mon[53087]: failed to encode map e115 with expected crc 2024-03-26T02:02:22.066 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:21 smithi019 ceph-mon[66359]: pgmap v160: 97 pgs: 1 active+recovery_wait+degraded, 2 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 92 active+clean; 338 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.6 MiB/s wr, 446 op/s; 12/1617 objects degraded (0.742%); 1.2 MiB/s, 34 objects/s recovering 2024-03-26T02:02:22.122 INFO:teuthology.orchestra.run.smithi019.stdout: 5 239259 10.65 MB/sec execute 415 sec latency 65.277 ms 2024-03-26T02:02:22.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:21 smithi069 ceph-mon[53087]: pgmap v160: 97 pgs: 1 active+recovery_wait+degraded, 2 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 92 active+clean; 338 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.6 MiB/s wr, 446 op/s; 12/1617 objects degraded (0.742%); 1.2 MiB/s, 34 objects/s recovering 2024-03-26T02:02:23.080 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:22 smithi019 ceph-mon[66359]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/1617 objects degraded (0.742%), 2 pgs degraded) 2024-03-26T02:02:23.081 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:22 smithi019 ceph-mon[66359]: Cluster is now healthy 2024-03-26T02:02:23.122 INFO:teuthology.orchestra.run.smithi019.stdout: 5 239608 10.64 MB/sec execute 416 sec latency 37.455 ms 2024-03-26T02:02:23.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:22 smithi069 ceph-mon[53087]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/1617 objects degraded (0.742%), 2 pgs degraded) 2024-03-26T02:02:23.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:22 smithi069 ceph-mon[53087]: Cluster is now healthy 2024-03-26T02:02:24.103 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:23 smithi019 ceph-mon[66359]: pgmap v161: 97 pgs: 97 active+clean; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.3 MiB/s wr, 383 op/s; 176 KiB/s, 16 objects/s recovering 2024-03-26T02:02:24.122 INFO:teuthology.orchestra.run.smithi019.stdout: 5 239959 10.65 MB/sec execute 417 sec latency 35.391 ms 2024-03-26T02:02:24.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:23 smithi069 ceph-mon[53087]: pgmap v161: 97 pgs: 97 active+clean; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.3 MiB/s wr, 383 op/s; 176 KiB/s, 16 objects/s recovering 2024-03-26T02:02:25.122 INFO:teuthology.orchestra.run.smithi019.stdout: 5 240302 10.65 MB/sec execute 418 sec latency 54.631 ms 2024-03-26T02:02:25.390 INFO:teuthology.orchestra.run.smithi019.stdout:true 2024-03-26T02:02:26.122 INFO:teuthology.orchestra.run.smithi019.stdout: 5 240595 10.64 MB/sec execute 419 sec latency 174.430 ms 2024-03-26T02:02:26.148 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:25 smithi019 ceph-mon[66359]: pgmap v162: 97 pgs: 97 active+clean; 339 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.8 MiB/s wr, 456 op/s; 38 KiB/s, 9 objects/s recovering 2024-03-26T02:02:26.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:25 smithi069 ceph-mon[53087]: pgmap v162: 97 pgs: 97 active+clean; 339 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.8 MiB/s wr, 456 op/s; 38 KiB/s, 9 objects/s recovering 2024-03-26T02:02:26.544 INFO:teuthology.orchestra.run.smithi019.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:02:26.544 INFO:teuthology.orchestra.run.smithi019.stdout:alertmanager.smithi019 smithi019 *:9093,9094 running (6m) 14s ago 15m 14.4M - 0.23.0 ba2b418f427c a6630841c2b2 2024-03-26T02:02:26.545 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi019 smithi019 running (2m) 14s ago 15m 6660k - 19.0.0-1775-g5e70124b 5f21c9264136 c5f91a47ddf2 2024-03-26T02:02:26.545 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi069 smithi069 running (2m) 2m ago 14m 6669k - 19.0.0-1775-g5e70124b 5f21c9264136 05128803b49e 2024-03-26T02:02:26.545 INFO:teuthology.orchestra.run.smithi019.stdout:grafana.smithi019 smithi019 *:3000 running (6m) 14s ago 15m 47.3M - 8.3.5 dad864ee21e9 a97dea3b5fec 2024-03-26T02:02:26.545 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi019.lmryom smithi019 running (9m) 14s ago 9m 38.3M - 17.2.0 e1d6a67b021e 91a7c2942715 2024-03-26T02:02:26.545 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi069.spgwgo smithi069 running (9m) 2m ago 9m 14.4M - 17.2.0 e1d6a67b021e 94d96338dbe1 2024-03-26T02:02:26.545 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi019.ixuvdm smithi019 *:8443,9283,8765 running (5m) 14s ago 16m 504M - 19.0.0-1775-g5e70124b 5f21c9264136 4460b22c7a56 2024-03-26T02:02:26.545 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi069.wpbewu smithi069 *:8443,9283,8765 running (4m) 2m ago 14m 451M - 19.0.0-1775-g5e70124b 5f21c9264136 380fd33842a8 2024-03-26T02:02:26.545 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi019 smithi019 running (4m) 14s ago 16m 50.2M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 c18934705774 2024-03-26T02:02:26.545 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi069 smithi069 running (3m) 2m ago 14m 35.6M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 a856bd263104 2024-03-26T02:02:26.545 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.0.0.smithi019.qbizxo smithi019 *:2049 running (9m) 14s ago 9m 283M - 4.0 e1d6a67b021e cd6628a9075d 2024-03-26T02:02:26.545 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.1.0.smithi069.uawdhc smithi069 *:2049 running (9m) 2m ago 9m 85.3M - 4.0 e1d6a67b021e 6ee3695e7a10 2024-03-26T02:02:26.545 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi019 smithi019 *:9100 running (5m) 14s ago 15m 10.2M - 1.3.1 1dbe0e931976 0d1d184bec63 2024-03-26T02:02:26.545 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi069 smithi069 *:9100 running (5m) 2m ago 14m 19.2M - 1.3.1 1dbe0e931976 c49a006f94e7 2024-03-26T02:02:26.546 INFO:teuthology.orchestra.run.smithi019.stdout:osd.0 smithi019 running (2m) 14s ago 14m 309M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 3f931cfe320a 2024-03-26T02:02:26.546 INFO:teuthology.orchestra.run.smithi019.stdout:osd.1 smithi019 running (72s) 14s ago 13m 249M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 758caf4ed841 2024-03-26T02:02:26.546 INFO:teuthology.orchestra.run.smithi019.stdout:osd.2 smithi019 running (24s) 14s ago 13m 12.6M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 6a9b701517ca 2024-03-26T02:02:26.546 INFO:teuthology.orchestra.run.smithi019.stdout:osd.3 smithi019 running (12m) 14s ago 12m 419M 1937M 17.2.0 e1d6a67b021e 154af68d03e1 2024-03-26T02:02:26.546 INFO:teuthology.orchestra.run.smithi019.stdout:osd.4 smithi069 running (12m) 2m ago 12m 556M 2704M 17.2.0 e1d6a67b021e 4fd5e853f8bf 2024-03-26T02:02:26.546 INFO:teuthology.orchestra.run.smithi019.stdout:osd.5 smithi069 running (11m) 2m ago 11m 433M 2704M 17.2.0 e1d6a67b021e 932e401d87c9 2024-03-26T02:02:26.546 INFO:teuthology.orchestra.run.smithi019.stdout:osd.6 smithi069 running (11m) 2m ago 11m 345M 2704M 17.2.0 e1d6a67b021e b123f75154c8 2024-03-26T02:02:26.546 INFO:teuthology.orchestra.run.smithi019.stdout:osd.7 smithi069 running (11m) 2m ago 11m 550M 2704M 17.2.0 e1d6a67b021e d175fbc9ff8a 2024-03-26T02:02:26.546 INFO:teuthology.orchestra.run.smithi019.stdout:prometheus.smithi019 smithi019 *:9095 running (4m) 14s ago 14m 67.5M - 2.33.4 514e6a882f6e 29f89db72a07 2024-03-26T02:02:27.115 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:02:27.115 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": { 2024-03-26T02:02:27.116 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:02:27.116 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:02:27.116 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": { 2024-03-26T02:02:27.116 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:02:27.116 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:02:27.116 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": { 2024-03-26T02:02:27.116 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-26T02:02:27.116 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 3 2024-03-26T02:02:27.116 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:02:27.116 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": { 2024-03-26T02:02:27.116 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:02:27.116 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:02:27.116 INFO:teuthology.orchestra.run.smithi019.stdout: "overall": { 2024-03-26T02:02:27.117 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-03-26T02:02:27.117 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 7 2024-03-26T02:02:27.117 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-26T02:02:27.117 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:02:27.122 INFO:teuthology.orchestra.run.smithi019.stdout: 5 240868 10.64 MB/sec execute 420 sec latency 53.524 ms 2024-03-26T02:02:27.156 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:26 smithi019 ceph-mon[66359]: from='client.34294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:02:27.157 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:26 smithi019 ceph-mon[66359]: from='client.34298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:02:27.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:26 smithi069 ceph-mon[53087]: from='client.34294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:02:27.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:26 smithi069 ceph-mon[53087]: from='client.34298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:02:27.626 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:02:27.626 INFO:teuthology.orchestra.run.smithi019.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2", 2024-03-26T02:02:27.626 INFO:teuthology.orchestra.run.smithi019.stdout: "in_progress": true, 2024-03-26T02:02:27.626 INFO:teuthology.orchestra.run.smithi019.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T02:02:27.626 INFO:teuthology.orchestra.run.smithi019.stdout: "services_complete": [ 2024-03-26T02:02:27.626 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr", 2024-03-26T02:02:27.627 INFO:teuthology.orchestra.run.smithi019.stdout: "crash", 2024-03-26T02:02:27.627 INFO:teuthology.orchestra.run.smithi019.stdout: "mon" 2024-03-26T02:02:27.627 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-26T02:02:27.627 INFO:teuthology.orchestra.run.smithi019.stdout: "progress": "9/23 daemons upgraded", 2024-03-26T02:02:27.627 INFO:teuthology.orchestra.run.smithi019.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T02:02:27.627 INFO:teuthology.orchestra.run.smithi019.stdout: "is_paused": false 2024-03-26T02:02:27.627 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:02:28.122 INFO:teuthology.orchestra.run.smithi019.stdout: 5 241151 10.63 MB/sec execute 421 sec latency 37.870 ms 2024-03-26T02:02:28.180 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:27 smithi019 ceph-mon[66359]: from='client.34302 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:02:28.181 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:27 smithi019 ceph-mon[66359]: pgmap v163: 97 pgs: 97 active+clean; 337 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.5 MiB/s wr, 596 op/s; 34 KiB/s, 8 objects/s recovering 2024-03-26T02:02:28.181 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:27 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/3511980355' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:02:28.251 INFO:teuthology.orchestra.run.smithi019.stdout:HEALTH_OK 2024-03-26T02:02:28.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:27 smithi069 ceph-mon[53087]: from='client.34302 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:02:28.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:27 smithi069 ceph-mon[53087]: pgmap v163: 97 pgs: 97 active+clean; 337 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.5 MiB/s wr, 596 op/s; 34 KiB/s, 8 objects/s recovering 2024-03-26T02:02:28.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:27 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/3511980355' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:02:29.122 INFO:teuthology.orchestra.run.smithi019.stdout: 5 241444 10.61 MB/sec execute 422 sec latency 62.434 ms 2024-03-26T02:02:29.195 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:28 smithi019 ceph-mon[66359]: from='client.34310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:02:29.196 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:28 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/78240633' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:02:29.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:28 smithi069 ceph-mon[53087]: from='client.34310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:02:29.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:28 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/78240633' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:02:30.123 INFO:teuthology.orchestra.run.smithi019.stdout: 5 241727 10.61 MB/sec execute 423 sec latency 56.651 ms 2024-03-26T02:02:30.205 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:29 smithi019 ceph-mon[66359]: pgmap v164: 97 pgs: 97 active+clean; 337 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.1 MiB/s wr, 511 op/s; 28 KiB/s, 7 objects/s recovering 2024-03-26T02:02:30.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:29 smithi069 ceph-mon[53087]: pgmap v164: 97 pgs: 97 active+clean; 337 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.1 MiB/s wr, 511 op/s; 28 KiB/s, 7 objects/s recovering 2024-03-26T02:02:31.122 INFO:teuthology.orchestra.run.smithi019.stdout: 5 242001 10.60 MB/sec execute 424 sec latency 74.185 ms 2024-03-26T02:02:32.002 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:31 smithi019 ceph-mon[66359]: pgmap v165: 97 pgs: 97 active+clean; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.9 MiB/s wr, 521 op/s; 25 KiB/s, 6 objects/s recovering 2024-03-26T02:02:32.003 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:31 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:32.003 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:31 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:02:32.003 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:31 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:31 smithi069 ceph-mon[53087]: pgmap v165: 97 pgs: 97 active+clean; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.9 MiB/s wr, 521 op/s; 25 KiB/s, 6 objects/s recovering 2024-03-26T02:02:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:31 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:31 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:02:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:31 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:32.122 INFO:teuthology.orchestra.run.smithi019.stdout: 5 242277 10.59 MB/sec execute 425 sec latency 75.363 ms 2024-03-26T02:02:33.123 INFO:teuthology.orchestra.run.smithi019.stdout: 5 242569 10.59 MB/sec execute 426 sec latency 55.678 ms 2024-03-26T02:02:33.800 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:33 smithi019 ceph-mon[66359]: pgmap v166: 97 pgs: 97 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.7 MiB/s wr, 571 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T02:02:34.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:33 smithi069 ceph-mon[53087]: pgmap v166: 97 pgs: 97 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.7 MiB/s wr, 571 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T02:02:34.122 INFO:teuthology.orchestra.run.smithi019.stdout: 5 242780 10.57 MB/sec execute 427 sec latency 81.455 ms 2024-03-26T02:02:34.829 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:34 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T02:02:35.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:34 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T02:02:35.123 INFO:teuthology.orchestra.run.smithi019.stdout: 5 243020 10.56 MB/sec execute 428 sec latency 61.924 ms 2024-03-26T02:02:35.849 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:35 smithi019 ceph-mon[66359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T02:02:35.849 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:35 smithi019 ceph-mon[66359]: Upgrade: osd.3 is safe to restart 2024-03-26T02:02:35.849 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:35 smithi019 ceph-mon[66359]: pgmap v167: 97 pgs: 97 active+clean; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.1 MiB/s wr, 430 op/s 2024-03-26T02:02:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:35 smithi069 ceph-mon[53087]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T02:02:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:35 smithi069 ceph-mon[53087]: Upgrade: osd.3 is safe to restart 2024-03-26T02:02:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:35 smithi069 ceph-mon[53087]: pgmap v167: 97 pgs: 97 active+clean; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.1 MiB/s wr, 430 op/s 2024-03-26T02:02:36.123 INFO:teuthology.orchestra.run.smithi019.stdout: 5 243288 10.56 MB/sec execute 429 sec latency 84.340 ms 2024-03-26T02:02:37.123 INFO:teuthology.orchestra.run.smithi019.stdout: 5 243649 10.56 MB/sec execute 430 sec latency 38.932 ms 2024-03-26T02:02:38.123 INFO:teuthology.orchestra.run.smithi019.stdout: 5 243948 10.56 MB/sec execute 431 sec latency 70.492 ms 2024-03-26T02:02:38.179 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:38 smithi019 ceph-mon[66359]: pgmap v168: 97 pgs: 97 active+clean; 338 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.0 MiB/s wr, 448 op/s 2024-03-26T02:02:38.179 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:38 smithi019 ceph-mon[66359]: Upgrade: Updating osd.3 2024-03-26T02:02:38.179 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:38 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:38.179 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:38 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T02:02:38.179 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:38 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:02:38.179 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:38 smithi019 ceph-mon[66359]: Deploying daemon osd.3 on smithi019 2024-03-26T02:02:38.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:38 smithi069 ceph-mon[53087]: pgmap v168: 97 pgs: 97 active+clean; 338 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.0 MiB/s wr, 448 op/s 2024-03-26T02:02:38.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:38 smithi069 ceph-mon[53087]: Upgrade: Updating osd.3 2024-03-26T02:02:38.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:38 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:38.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:38 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T02:02:38.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:38 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:02:38.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:38 smithi069 ceph-mon[53087]: Deploying daemon osd.3 on smithi019 2024-03-26T02:02:39.123 INFO:teuthology.orchestra.run.smithi019.stdout: 5 244239 10.55 MB/sec execute 432 sec latency 41.643 ms 2024-03-26T02:02:40.123 INFO:teuthology.orchestra.run.smithi019.stdout: 5 244678 10.55 MB/sec execute 433 sec latency 32.619 ms 2024-03-26T02:02:40.210 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:40 smithi019 ceph-mon[66359]: pgmap v169: 97 pgs: 97 active+clean; 338 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.7 MiB/s wr, 327 op/s 2024-03-26T02:02:40.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:40 smithi069 ceph-mon[53087]: pgmap v169: 97 pgs: 97 active+clean; 338 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.7 MiB/s wr, 327 op/s 2024-03-26T02:02:41.123 INFO:teuthology.orchestra.run.smithi019.stdout: 5 245070 10.54 MB/sec execute 434 sec latency 35.613 ms 2024-03-26T02:02:42.123 INFO:teuthology.orchestra.run.smithi019.stdout: 5 245468 10.55 MB/sec execute 435 sec latency 48.114 ms 2024-03-26T02:02:42.227 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:42 smithi019 ceph-mon[66359]: pgmap v170: 97 pgs: 97 active+clean; 338 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.7 MiB/s wr, 407 op/s 2024-03-26T02:02:42.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:42 smithi069 ceph-mon[53087]: pgmap v170: 97 pgs: 97 active+clean; 338 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.7 MiB/s wr, 407 op/s 2024-03-26T02:02:42.761 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:42 smithi019 systemd[1]: Stopping Ceph osd.3 for 692607c6-eb12-11ee-95d0-87774f69a715... 2024-03-26T02:02:43.123 INFO:teuthology.orchestra.run.smithi019.stdout: 5 245807 10.54 MB/sec execute 436 sec latency 87.770 ms 2024-03-26T02:02:43.263 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:43 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3[46353]: 2024-03-26T02:02:43.045+0000 7fcf71b2b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:02:43.264 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:43 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3[46353]: 2024-03-26T02:02:43.045+0000 7fcf71b2b700 -1 osd.3 115 *** Got signal Terminated *** 2024-03-26T02:02:43.264 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:43 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3[46353]: 2024-03-26T02:02:43.045+0000 7fcf71b2b700 -1 osd.3 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:02:44.024 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:44 smithi019 ceph-mon[66359]: pgmap v171: 97 pgs: 97 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.2 MiB/s wr, 496 op/s 2024-03-26T02:02:44.024 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:44 smithi019 ceph-mon[66359]: osd.3 marked itself down and dead 2024-03-26T02:02:44.123 INFO:teuthology.orchestra.run.smithi019.stdout: 5 246128 10.54 MB/sec execute 437 sec latency 105.523 ms 2024-03-26T02:02:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:44 smithi069 ceph-mon[53087]: pgmap v171: 97 pgs: 97 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.2 MiB/s wr, 496 op/s 2024-03-26T02:02:44.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:44 smithi069 ceph-mon[53087]: osd.3 marked itself down and dead 2024-03-26T02:02:44.982 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:44 smithi019 podman[79890]: 2024-03-26 02:02:44.704131916 +0000 UTC m=+1.810439247 container died 154af68d03e1d1360a9a70b55ed8e12e3552baca838c89aab3e75fe514795185 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, vcs-type=git, RELEASE=HEAD, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, release=754, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, GIT_CLEAN=True, vendor=Red Hat, Inc., io.openshift.expose-services=) 2024-03-26T02:02:45.123 INFO:teuthology.orchestra.run.smithi019.stdout: 5 246133 10.52 MB/sec execute 438 sec latency 1086.167 ms 2024-03-26T02:02:45.235 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:45 smithi019 podman[79890]: 2024-03-26 02:02:45.182627771 +0000 UTC m=+2.288935089 container cleanup 154af68d03e1d1360a9a70b55ed8e12e3552baca838c89aab3e75fe514795185 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, version=8, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, name=centos-stream, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, ceph=True, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-03-26T02:02:45.235 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:45 smithi019 bash[79890]: ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3 2024-03-26T02:02:45.235 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:45 smithi019 ceph-mon[66359]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:02:45.235 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:45 smithi019 ceph-mon[66359]: failed to encode map e116 with expected crc 2024-03-26T02:02:45.235 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:45 smithi019 ceph-mon[66359]: osdmap e116: 8 total, 7 up, 8 in 2024-03-26T02:02:45.235 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:45 smithi019 ceph-mon[66359]: failed to encode map e116 with expected crc 2024-03-26T02:02:45.235 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:45 smithi019 ceph-mon[66359]: failed to encode map e116 with expected crc 2024-03-26T02:02:45.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:45 smithi069 ceph-mon[53087]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:02:45.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:45 smithi069 ceph-mon[53087]: failed to encode map e116 with expected crc 2024-03-26T02:02:45.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:45 smithi069 ceph-mon[53087]: osdmap e116: 8 total, 7 up, 8 in 2024-03-26T02:02:45.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:45 smithi069 ceph-mon[53087]: failed to encode map e116 with expected crc 2024-03-26T02:02:45.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:45 smithi069 ceph-mon[53087]: failed to encode map e116 with expected crc 2024-03-26T02:02:45.994 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:45 smithi019 podman[79905]: 2024-03-26 02:02:45.956258658 +0000 UTC m=+1.249571109 container remove 154af68d03e1d1360a9a70b55ed8e12e3552baca838c89aab3e75fe514795185 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, vcs-type=git, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, release=754, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True) 2024-03-26T02:02:46.123 INFO:teuthology.orchestra.run.smithi019.stdout: 5 246554 10.51 MB/sec execute 439 sec latency 33.039 ms 2024-03-26T02:02:46.245 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:46 smithi019 ceph-mon[66359]: pgmap v173: 97 pgs: 9 peering, 12 stale+active+clean, 76 active+clean; 344 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.4 KiB/s rd, 4.7 MiB/s wr, 457 op/s 2024-03-26T02:02:46.245 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:46 smithi019 ceph-mon[66359]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-26T02:02:46.245 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:46 smithi019 ceph-mon[66359]: failed to encode map e117 with expected crc 2024-03-26T02:02:46.245 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:46 smithi019 ceph-mon[66359]: osdmap e117: 8 total, 7 up, 8 in 2024-03-26T02:02:46.245 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:46 smithi019 ceph-mon[66359]: failed to encode map e117 with expected crc 2024-03-26T02:02:46.246 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:46 smithi019 ceph-mon[66359]: failed to encode map e117 with expected crc 2024-03-26T02:02:46.246 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:46 smithi019 ceph-mon[66359]: failed to encode map e117 with expected crc 2024-03-26T02:02:46.246 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:46 smithi019 ceph-mon[66359]: failed to encode map e117 with expected crc 2024-03-26T02:02:46.246 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:46 smithi019 ceph-mon[66359]: failed to encode map e117 with expected crc 2024-03-26T02:02:46.246 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:46 smithi019 ceph-mon[66359]: failed to encode map e117 with expected crc 2024-03-26T02:02:46.246 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:46 smithi019 ceph-mon[66359]: failed to encode map e117 with expected crc 2024-03-26T02:02:46.246 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:46 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:46.246 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:46 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:02:46.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:46 smithi069 ceph-mon[53087]: pgmap v173: 97 pgs: 9 peering, 12 stale+active+clean, 76 active+clean; 344 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.4 KiB/s rd, 4.7 MiB/s wr, 457 op/s 2024-03-26T02:02:46.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:46 smithi069 ceph-mon[53087]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-26T02:02:46.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:46 smithi069 ceph-mon[53087]: failed to encode map e117 with expected crc 2024-03-26T02:02:46.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:46 smithi069 ceph-mon[53087]: osdmap e117: 8 total, 7 up, 8 in 2024-03-26T02:02:46.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:46 smithi069 ceph-mon[53087]: failed to encode map e117 with expected crc 2024-03-26T02:02:46.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:46 smithi069 ceph-mon[53087]: failed to encode map e117 with expected crc 2024-03-26T02:02:46.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:46 smithi069 ceph-mon[53087]: failed to encode map e117 with expected crc 2024-03-26T02:02:46.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:46 smithi069 ceph-mon[53087]: failed to encode map e117 with expected crc 2024-03-26T02:02:46.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:46 smithi069 ceph-mon[53087]: failed to encode map e117 with expected crc 2024-03-26T02:02:46.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:46 smithi069 ceph-mon[53087]: failed to encode map e117 with expected crc 2024-03-26T02:02:46.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:46 smithi069 ceph-mon[53087]: failed to encode map e117 with expected crc 2024-03-26T02:02:46.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:46 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:46.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:46 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:02:46.500 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:46 smithi019 podman[80025]: 2024-03-26 02:02:46.29019067 +0000 UTC m=+0.024625100 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:02:47.031 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:47 smithi019 ceph-mon[66359]: failed to encode map e117 with expected crc 2024-03-26T02:02:47.032 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:47 smithi019 ceph-mon[66359]: failed to encode map e117 with expected crc 2024-03-26T02:02:47.032 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:47 smithi019 ceph-mon[66359]: failed to encode map e117 with expected crc 2024-03-26T02:02:47.032 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:46 smithi019 podman[80025]: 2024-03-26 02:02:46.948307875 +0000 UTC m=+0.682742289 container create a712dd09facc567f974adadd2ffbbc985620ac455f5f9ecb59877c1b014e853c (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:02:47.123 INFO:teuthology.orchestra.run.smithi019.stdout: 5 246917 10.52 MB/sec execute 440 sec latency 39.973 ms 2024-03-26T02:02:47.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:47 smithi069 ceph-mon[53087]: failed to encode map e117 with expected crc 2024-03-26T02:02:47.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:47 smithi069 ceph-mon[53087]: failed to encode map e117 with expected crc 2024-03-26T02:02:47.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:47 smithi069 ceph-mon[53087]: failed to encode map e117 with expected crc 2024-03-26T02:02:47.418 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:47 smithi019 podman[80025]: 2024-03-26 02:02:47.335843666 +0000 UTC m=+1.070278092 container init a712dd09facc567f974adadd2ffbbc985620ac455f5f9ecb59877c1b014e853c (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3-deactivate, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-26T02:02:47.419 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:47 smithi019 podman[80025]: 2024-03-26 02:02:47.348656251 +0000 UTC m=+1.083090673 container start a712dd09facc567f974adadd2ffbbc985620ac455f5f9ecb59877c1b014e853c (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3-deactivate, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124) 2024-03-26T02:02:47.669 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:47 smithi019 podman[80025]: 2024-03-26 02:02:47.511608574 +0000 UTC m=+1.246043017 container attach a712dd09facc567f974adadd2ffbbc985620ac455f5f9ecb59877c1b014e853c (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-26T02:02:47.928 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:47 smithi019 podman[80025]: 2024-03-26 02:02:47.809293948 +0000 UTC m=+1.543728372 container died a712dd09facc567f974adadd2ffbbc985620ac455f5f9ecb59877c1b014e853c (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-26T02:02:48.123 INFO:teuthology.orchestra.run.smithi019.stdout: 5 247292 10.52 MB/sec execute 441 sec latency 53.881 ms 2024-03-26T02:02:48.188 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:48 smithi019 ceph-mon[66359]: pgmap v175: 97 pgs: 9 active+undersized, 14 active+undersized+degraded, 9 peering, 3 stale+active+clean, 62 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.5 MiB/s wr, 589 op/s; 114/1653 objects degraded (6.897%) 2024-03-26T02:02:48.188 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:48 smithi019 ceph-mon[66359]: Health check failed: Degraded data redundancy: 114/1653 objects degraded (6.897%), 14 pgs degraded (PG_DEGRADED) 2024-03-26T02:02:48.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:48 smithi069 ceph-mon[53087]: pgmap v175: 97 pgs: 9 active+undersized, 14 active+undersized+degraded, 9 peering, 3 stale+active+clean, 62 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.5 MiB/s wr, 589 op/s; 114/1653 objects degraded (6.897%) 2024-03-26T02:02:48.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:48 smithi069 ceph-mon[53087]: Health check failed: Degraded data redundancy: 114/1653 objects degraded (6.897%), 14 pgs degraded (PG_DEGRADED) 2024-03-26T02:02:49.123 INFO:teuthology.orchestra.run.smithi019.stdout: 5 247700 10.52 MB/sec execute 442 sec latency 41.344 ms 2024-03-26T02:02:49.199 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:49 smithi019 podman[80051]: 2024-03-26 02:02:49.052335095 +0000 UTC m=+1.235030927 container remove a712dd09facc567f974adadd2ffbbc985620ac455f5f9ecb59877c1b014e853c (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-26T02:02:49.199 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:49 smithi019 systemd[1]: ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.3.service: Deactivated successfully. 2024-03-26T02:02:49.199 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:49 smithi019 systemd[1]: Stopped Ceph osd.3 for 692607c6-eb12-11ee-95d0-87774f69a715. 2024-03-26T02:02:49.199 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:49 smithi019 systemd[1]: ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.3.service: Consumed 5min 10.158s CPU time. 2024-03-26T02:02:49.462 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:49 smithi019 systemd[1]: Starting Ceph osd.3 for 692607c6-eb12-11ee-95d0-87774f69a715... 2024-03-26T02:02:49.712 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:49 smithi019 podman[80150]: 2024-03-26 02:02:49.552580011 +0000 UTC m=+0.019772194 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:02:49.713 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:49 smithi019 podman[80150]: 2024-03-26 02:02:49.658009054 +0000 UTC m=+0.125201216 container create 95b2f7db9c2c5f7bae42d91d5d7f8b7896522b0d8f9824e65ecee79d53edbbd8 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3-activate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124) 2024-03-26T02:02:49.963 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:49 smithi019 podman[80150]: 2024-03-26 02:02:49.945059757 +0000 UTC m=+0.412251930 container init 95b2f7db9c2c5f7bae42d91d5d7f8b7896522b0d8f9824e65ecee79d53edbbd8 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-26T02:02:49.964 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:49 smithi019 podman[80150]: 2024-03-26 02:02:49.951163091 +0000 UTC m=+0.418355253 container start 95b2f7db9c2c5f7bae42d91d5d7f8b7896522b0d8f9824e65ecee79d53edbbd8 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3-activate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:02:50.123 INFO:teuthology.orchestra.run.smithi019.stdout: 5 247916 10.51 MB/sec execute 443 sec latency 62.095 ms 2024-03-26T02:02:50.215 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:50 smithi019 ceph-mon[66359]: pgmap v176: 97 pgs: 11 active+undersized, 21 active+undersized+degraded, 9 peering, 56 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.3 MiB/s wr, 471 op/s; 186/1656 objects degraded (11.232%) 2024-03-26T02:02:50.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:50 smithi069 ceph-mon[53087]: pgmap v176: 97 pgs: 11 active+undersized, 21 active+undersized+degraded, 9 peering, 56 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.3 MiB/s wr, 471 op/s; 186/1656 objects degraded (11.232%) 2024-03-26T02:02:50.486 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:50 smithi019 podman[80150]: 2024-03-26 02:02:50.253956863 +0000 UTC m=+0.721149027 container attach 95b2f7db9c2c5f7bae42d91d5d7f8b7896522b0d8f9824e65ecee79d53edbbd8 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True) 2024-03-26T02:02:50.487 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:50 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3-activate[80165]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-26T02:02:50.487 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:50 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3-activate[80165]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-26T02:02:50.487 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:50 smithi019 bash[80150]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-26T02:02:50.487 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:50 smithi019 bash[80150]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-26T02:02:50.487 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:50 smithi019 bash[80150]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-26T02:02:50.487 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:50 smithi019 bash[80150]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-26T02:02:50.487 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:50 smithi019 bash[80150]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-26T02:02:50.487 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:50 smithi019 bash[80150]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-26T02:02:50.487 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:50 smithi019 bash[80150]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-26T02:02:50.488 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:50 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3-activate[80165]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-26T02:02:50.488 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:50 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3-activate[80165]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-26T02:02:50.488 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:50 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3-activate[80165]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-26T02:02:50.488 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:50 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3-activate[80165]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-26T02:02:50.488 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:50 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3-activate[80165]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-26T02:02:50.742 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:50 smithi019 podman[80150]: 2024-03-26 02:02:50.504312875 +0000 UTC m=+0.971505051 container died 95b2f7db9c2c5f7bae42d91d5d7f8b7896522b0d8f9824e65ecee79d53edbbd8 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3-activate, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=) 2024-03-26T02:02:51.124 INFO:teuthology.orchestra.run.smithi019.stdout: 5 248160 10.50 MB/sec execute 444 sec latency 58.929 ms 2024-03-26T02:02:51.244 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:51 smithi019 ceph-mon[66359]: pgmap v177: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 335 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 2.9 MiB/s wr, 258 op/s; 217/1653 objects degraded (13.128%) 2024-03-26T02:02:51.245 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:51 smithi019 ceph-mon[66359]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-03-26T02:02:51.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:51 smithi069 ceph-mon[53087]: pgmap v177: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 335 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 2.9 MiB/s wr, 258 op/s; 217/1653 objects degraded (13.128%) 2024-03-26T02:02:51.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:51 smithi069 ceph-mon[53087]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-03-26T02:02:52.124 INFO:teuthology.orchestra.run.smithi019.stdout: 5 248421 10.49 MB/sec execute 445 sec latency 55.210 ms 2024-03-26T02:02:52.268 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:52 smithi019 podman[80287]: 2024-03-26 02:02:52.071867747 +0000 UTC m=+1.552411925 container remove 95b2f7db9c2c5f7bae42d91d5d7f8b7896522b0d8f9824e65ecee79d53edbbd8 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3-activate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-5e70124, org.label-schema.build-date=20240319) 2024-03-26T02:02:52.532 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:52 smithi019 podman[80340]: 2024-03-26 02:02:52.211084778 +0000 UTC m=+0.022845426 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:02:52.532 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:52 smithi019 podman[80340]: 2024-03-26 02:02:52.375209447 +0000 UTC m=+0.186970092 container create 0401940d882b660b6928430c55ee7f32363ad9c83db4acf89a63bf56c9bf60b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-26T02:02:52.783 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:52 smithi019 podman[80340]: 2024-03-26 02:02:52.737696515 +0000 UTC m=+0.549457152 container init 0401940d882b660b6928430c55ee7f32363ad9c83db4acf89a63bf56c9bf60b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-5e70124) 2024-03-26T02:02:52.783 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:52 smithi019 podman[80340]: 2024-03-26 02:02:52.743883343 +0000 UTC m=+0.555643988 container start 0401940d882b660b6928430c55ee7f32363ad9c83db4acf89a63bf56c9bf60b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3, RELEASE=squid-5e70124, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:02:52.783 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:52 smithi019 ceph-osd[80358]: -- 172.21.15.19:0/1161224759 <== mon.0 v2:172.21.15.19:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557f0f9791e0 con 0x557f0f94ac00 2024-03-26T02:02:53.039 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:52 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 220/1668 objects degraded (13.189%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T02:02:53.040 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:52 smithi019 bash[80340]: 0401940d882b660b6928430c55ee7f32363ad9c83db4acf89a63bf56c9bf60b4 2024-03-26T02:02:53.040 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:52 smithi019 systemd[1]: Started Ceph osd.3 for 692607c6-eb12-11ee-95d0-87774f69a715. 2024-03-26T02:02:53.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:52 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 220/1668 objects degraded (13.189%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T02:02:53.124 INFO:teuthology.orchestra.run.smithi019.stdout: 5 248690 10.48 MB/sec execute 446 sec latency 80.546 ms 2024-03-26T02:02:53.567 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:02:53 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3[80354]: 2024-03-26T02:02:53.320+0000 7f3b8ed117c0 -1 Falling back to public interface 2024-03-26T02:02:54.124 INFO:teuthology.orchestra.run.smithi019.stdout: 5 248922 10.47 MB/sec execute 447 sec latency 62.769 ms 2024-03-26T02:02:54.146 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:53 smithi019 ceph-mon[66359]: pgmap v178: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 5.2 MiB/s wr, 514 op/s; 220/1668 objects degraded (13.189%) 2024-03-26T02:02:54.147 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:53 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:54.147 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:53 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:54.147 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:53 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:02:54.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:53 smithi069 ceph-mon[53087]: pgmap v178: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 5.2 MiB/s wr, 514 op/s; 220/1668 objects degraded (13.189%) 2024-03-26T02:02:54.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:53 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:54.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:53 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:02:54.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:53 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:02:55.124 INFO:teuthology.orchestra.run.smithi019.stdout: 5 249285 10.47 MB/sec execute 448 sec latency 44.597 ms 2024-03-26T02:02:56.124 INFO:teuthology.orchestra.run.smithi019.stdout: 5 249614 10.46 MB/sec execute 449 sec latency 59.468 ms 2024-03-26T02:02:56.207 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:55 smithi019 ceph-mon[66359]: pgmap v179: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 333 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.4 MiB/s wr, 444 op/s; 218/1647 objects degraded (13.236%) 2024-03-26T02:02:56.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:55 smithi069 ceph-mon[53087]: pgmap v179: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 333 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.4 MiB/s wr, 444 op/s; 218/1647 objects degraded (13.236%) 2024-03-26T02:02:57.124 INFO:teuthology.orchestra.run.smithi019.stdout: 5 250008 10.46 MB/sec execute 450 sec latency 52.364 ms 2024-03-26T02:02:58.003 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:57 smithi019 ceph-mon[66359]: pgmap v180: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 338 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.3 MiB/s wr, 556 op/s; 219/1659 objects degraded (13.201%) 2024-03-26T02:02:58.124 INFO:teuthology.orchestra.run.smithi019.stdout: 5 250290 10.46 MB/sec execute 451 sec latency 35.325 ms 2024-03-26T02:02:58.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:57 smithi069 ceph-mon[53087]: pgmap v180: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 338 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.3 MiB/s wr, 556 op/s; 219/1659 objects degraded (13.201%) 2024-03-26T02:02:58.750 INFO:teuthology.orchestra.run.smithi019.stdout:true 2024-03-26T02:02:59.022 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:02:59 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 220/1647 objects degraded (13.358%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T02:02:59.124 INFO:teuthology.orchestra.run.smithi019.stdout: 5 250652 10.47 MB/sec execute 452 sec latency 52.508 ms 2024-03-26T02:02:59.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:02:59 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 220/1647 objects degraded (13.358%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T02:02:59.686 INFO:teuthology.orchestra.run.smithi019.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:02:59.687 INFO:teuthology.orchestra.run.smithi019.stdout:alertmanager.smithi019 smithi019 *:9093,9094 running (6m) 47s ago 16m 14.4M - 0.23.0 ba2b418f427c a6630841c2b2 2024-03-26T02:02:59.687 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi019 smithi019 running (3m) 47s ago 16m 6660k - 19.0.0-1775-g5e70124b 5f21c9264136 c5f91a47ddf2 2024-03-26T02:02:59.687 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi069 smithi069 running (3m) 3m ago 15m 6669k - 19.0.0-1775-g5e70124b 5f21c9264136 05128803b49e 2024-03-26T02:02:59.687 INFO:teuthology.orchestra.run.smithi019.stdout:grafana.smithi019 smithi019 *:3000 running (6m) 47s ago 15m 47.3M - 8.3.5 dad864ee21e9 a97dea3b5fec 2024-03-26T02:02:59.687 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi019.lmryom smithi019 running (10m) 47s ago 10m 38.3M - 17.2.0 e1d6a67b021e 91a7c2942715 2024-03-26T02:02:59.687 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi069.spgwgo smithi069 running (10m) 3m ago 10m 14.4M - 17.2.0 e1d6a67b021e 94d96338dbe1 2024-03-26T02:02:59.687 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi019.ixuvdm smithi019 *:8443,9283,8765 running (5m) 47s ago 17m 504M - 19.0.0-1775-g5e70124b 5f21c9264136 4460b22c7a56 2024-03-26T02:02:59.687 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi069.wpbewu smithi069 *:8443,9283,8765 running (4m) 3m ago 15m 451M - 19.0.0-1775-g5e70124b 5f21c9264136 380fd33842a8 2024-03-26T02:02:59.687 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi019 smithi019 running (4m) 47s ago 17m 50.2M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 c18934705774 2024-03-26T02:02:59.687 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi069 smithi069 running (4m) 3m ago 15m 35.6M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 a856bd263104 2024-03-26T02:02:59.687 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.0.0.smithi019.qbizxo smithi019 *:2049 running (9m) 47s ago 9m 283M - 4.0 e1d6a67b021e cd6628a9075d 2024-03-26T02:02:59.687 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.1.0.smithi069.uawdhc smithi069 *:2049 running (9m) 3m ago 9m 85.3M - 4.0 e1d6a67b021e 6ee3695e7a10 2024-03-26T02:02:59.688 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi019 smithi019 *:9100 running (6m) 47s ago 15m 10.2M - 1.3.1 1dbe0e931976 0d1d184bec63 2024-03-26T02:02:59.688 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi069 smithi069 *:9100 running (6m) 3m ago 15m 19.2M - 1.3.1 1dbe0e931976 c49a006f94e7 2024-03-26T02:02:59.688 INFO:teuthology.orchestra.run.smithi019.stdout:osd.0 smithi019 running (2m) 47s ago 14m 309M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 3f931cfe320a 2024-03-26T02:02:59.688 INFO:teuthology.orchestra.run.smithi019.stdout:osd.1 smithi019 running (106s) 47s ago 14m 249M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 758caf4ed841 2024-03-26T02:02:59.688 INFO:teuthology.orchestra.run.smithi019.stdout:osd.2 smithi019 running (57s) 47s ago 13m 12.6M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 6a9b701517ca 2024-03-26T02:02:59.688 INFO:teuthology.orchestra.run.smithi019.stdout:osd.3 smithi019 starting - - - 1937M 2024-03-26T02:02:59.688 INFO:teuthology.orchestra.run.smithi019.stdout:osd.4 smithi069 running (12m) 3m ago 12m 556M 2704M 17.2.0 e1d6a67b021e 4fd5e853f8bf 2024-03-26T02:02:59.688 INFO:teuthology.orchestra.run.smithi019.stdout:osd.5 smithi069 running (12m) 3m ago 12m 433M 2704M 17.2.0 e1d6a67b021e 932e401d87c9 2024-03-26T02:02:59.688 INFO:teuthology.orchestra.run.smithi019.stdout:osd.6 smithi069 running (12m) 3m ago 11m 345M 2704M 17.2.0 e1d6a67b021e b123f75154c8 2024-03-26T02:02:59.688 INFO:teuthology.orchestra.run.smithi019.stdout:osd.7 smithi069 running (11m) 3m ago 11m 550M 2704M 17.2.0 e1d6a67b021e d175fbc9ff8a 2024-03-26T02:02:59.688 INFO:teuthology.orchestra.run.smithi019.stdout:prometheus.smithi019 smithi019 *:9095 running (5m) 47s ago 14m 67.5M - 2.33.4 514e6a882f6e 29f89db72a07 2024-03-26T02:03:00.042 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:00 smithi019 ceph-mon[66359]: pgmap v181: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 336 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.2 MiB/s wr, 416 op/s; 220/1647 objects degraded (13.358%) 2024-03-26T02:03:00.043 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:00 smithi019 ceph-mon[66359]: from='client.34320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:03:00.124 INFO:teuthology.orchestra.run.smithi019.stdout: 5 250971 10.46 MB/sec execute 453 sec latency 55.585 ms 2024-03-26T02:03:00.258 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:03:00.259 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": { 2024-03-26T02:03:00.259 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:03:00.259 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:03:00.259 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": { 2024-03-26T02:03:00.259 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:03:00.259 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:03:00.259 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": { 2024-03-26T02:03:00.259 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-03-26T02:03:00.259 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 3 2024-03-26T02:03:00.259 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:03:00.259 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": { 2024-03-26T02:03:00.260 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:03:00.260 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:03:00.260 INFO:teuthology.orchestra.run.smithi019.stdout: "overall": { 2024-03-26T02:03:00.260 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-03-26T02:03:00.260 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 7 2024-03-26T02:03:00.260 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-26T02:03:00.260 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:03:00.293 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:03:00 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3[80354]: 2024-03-26T02:03:00.165+0000 7f3b8ed117c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-03-26T02:03:00.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:00 smithi069 ceph-mon[53087]: pgmap v181: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 336 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.2 MiB/s wr, 416 op/s; 220/1647 objects degraded (13.358%) 2024-03-26T02:03:00.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:00 smithi069 ceph-mon[53087]: from='client.34320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:03:00.852 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:03:00.852 INFO:teuthology.orchestra.run.smithi019.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2", 2024-03-26T02:03:00.852 INFO:teuthology.orchestra.run.smithi019.stdout: "in_progress": true, 2024-03-26T02:03:00.852 INFO:teuthology.orchestra.run.smithi019.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T02:03:00.852 INFO:teuthology.orchestra.run.smithi019.stdout: "services_complete": [ 2024-03-26T02:03:00.852 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr", 2024-03-26T02:03:00.852 INFO:teuthology.orchestra.run.smithi019.stdout: "crash", 2024-03-26T02:03:00.852 INFO:teuthology.orchestra.run.smithi019.stdout: "mon" 2024-03-26T02:03:00.853 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-26T02:03:00.853 INFO:teuthology.orchestra.run.smithi019.stdout: "progress": "9/23 daemons upgraded", 2024-03-26T02:03:00.853 INFO:teuthology.orchestra.run.smithi019.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T02:03:00.853 INFO:teuthology.orchestra.run.smithi019.stdout: "is_paused": false 2024-03-26T02:03:00.853 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:03:01.047 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:01 smithi019 ceph-mon[66359]: from='client.44251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:03:01.047 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:01 smithi019 ceph-mon[66359]: from='client.34328 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:03:01.047 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:01 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/1614462442' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:03:01.047 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:01.047 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:03:01.124 INFO:teuthology.orchestra.run.smithi019.stdout: 5 251191 10.45 MB/sec execute 454 sec latency 47.725 ms 2024-03-26T02:03:01.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:01 smithi069 ceph-mon[53087]: from='client.44251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:03:01.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:01 smithi069 ceph-mon[53087]: from='client.34328 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:03:01.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:01 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/1614462442' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:03:01.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:01.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:03:01.572 INFO:teuthology.orchestra.run.smithi019.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 220/1662 objects degraded (13.237%), 27 pgs degraded 2024-03-26T02:03:01.572 INFO:teuthology.orchestra.run.smithi019.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-26T02:03:01.572 INFO:teuthology.orchestra.run.smithi019.stdout: osd.3 (root=default,host=smithi019) is down 2024-03-26T02:03:01.572 INFO:teuthology.orchestra.run.smithi019.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 220/1662 objects degraded (13.237%), 27 pgs degraded 2024-03-26T02:03:01.572 INFO:teuthology.orchestra.run.smithi019.stdout: pg 2.1 is active+undersized+degraded, acting [2,0] 2024-03-26T02:03:01.572 INFO:teuthology.orchestra.run.smithi019.stdout: pg 2.9 is active+undersized+degraded, acting [1,7] 2024-03-26T02:03:01.573 INFO:teuthology.orchestra.run.smithi019.stdout: pg 2.a is active+undersized+degraded, acting [1,7] 2024-03-26T02:03:01.573 INFO:teuthology.orchestra.run.smithi019.stdout: pg 2.d is active+undersized+degraded, acting [1,4] 2024-03-26T02:03:01.573 INFO:teuthology.orchestra.run.smithi019.stdout: pg 2.e is active+undersized+degraded, acting [2,7] 2024-03-26T02:03:01.573 INFO:teuthology.orchestra.run.smithi019.stdout: pg 2.12 is active+undersized+degraded, acting [5,7] 2024-03-26T02:03:01.573 INFO:teuthology.orchestra.run.smithi019.stdout: pg 2.13 is active+undersized+degraded, acting [0,4] 2024-03-26T02:03:01.573 INFO:teuthology.orchestra.run.smithi019.stdout: pg 2.14 is active+undersized+degraded, acting [6,5] 2024-03-26T02:03:01.573 INFO:teuthology.orchestra.run.smithi019.stdout: pg 2.19 is active+undersized+degraded, acting [6,0] 2024-03-26T02:03:01.573 INFO:teuthology.orchestra.run.smithi019.stdout: pg 2.1b is active+undersized+degraded, acting [7,6] 2024-03-26T02:03:01.573 INFO:teuthology.orchestra.run.smithi019.stdout: pg 2.1e is active+undersized+degraded, acting [0,5] 2024-03-26T02:03:01.573 INFO:teuthology.orchestra.run.smithi019.stdout: pg 3.1 is active+undersized+degraded, acting [0,4] 2024-03-26T02:03:01.573 INFO:teuthology.orchestra.run.smithi019.stdout: pg 3.2 is active+undersized+degraded, acting [5,6] 2024-03-26T02:03:01.573 INFO:teuthology.orchestra.run.smithi019.stdout: pg 3.5 is active+undersized+degraded, acting [5,2] 2024-03-26T02:03:01.573 INFO:teuthology.orchestra.run.smithi019.stdout: pg 3.7 is active+undersized+degraded, acting [7,0] 2024-03-26T02:03:01.573 INFO:teuthology.orchestra.run.smithi019.stdout: pg 3.8 is active+undersized+degraded, acting [1,7] 2024-03-26T02:03:01.573 INFO:teuthology.orchestra.run.smithi019.stdout: pg 3.b is active+undersized+degraded, acting [0,4] 2024-03-26T02:03:01.574 INFO:teuthology.orchestra.run.smithi019.stdout: pg 3.c is active+undersized+degraded, acting [5,6] 2024-03-26T02:03:01.574 INFO:teuthology.orchestra.run.smithi019.stdout: pg 3.12 is active+undersized+degraded, acting [0,7] 2024-03-26T02:03:01.574 INFO:teuthology.orchestra.run.smithi019.stdout: pg 3.15 is active+undersized+degraded, acting [7,4] 2024-03-26T02:03:01.574 INFO:teuthology.orchestra.run.smithi019.stdout: pg 3.17 is active+undersized+degraded, acting [0,5] 2024-03-26T02:03:01.574 INFO:teuthology.orchestra.run.smithi019.stdout: pg 3.18 is active+undersized+degraded, acting [0,1] 2024-03-26T02:03:01.574 INFO:teuthology.orchestra.run.smithi019.stdout: pg 3.19 is active+undersized+degraded, acting [1,4] 2024-03-26T02:03:01.574 INFO:teuthology.orchestra.run.smithi019.stdout: pg 3.1e is active+undersized+degraded, acting [6,2] 2024-03-26T02:03:01.574 INFO:teuthology.orchestra.run.smithi019.stdout: pg 4.0 is active+undersized+degraded, acting [7,0] 2024-03-26T02:03:01.574 INFO:teuthology.orchestra.run.smithi019.stdout: pg 4.14 is active+undersized+degraded, acting [1,7] 2024-03-26T02:03:01.574 INFO:teuthology.orchestra.run.smithi019.stdout: pg 4.19 is active+undersized+degraded, acting [2,0] 2024-03-26T02:03:02.091 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:03:02 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3[80354]: 2024-03-26T02:03:02.039+0000 7f3b8ed117c0 -1 osd.3 115 log_to_monitors true 2024-03-26T02:03:02.092 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:02 smithi019 ceph-mon[66359]: pgmap v182: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 338 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.0 MiB/s wr, 407 op/s; 220/1662 objects degraded (13.237%) 2024-03-26T02:03:02.092 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:02 smithi019 ceph-mon[66359]: from='client.44259 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:03:02.092 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:02 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/2673332040' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:03:02.124 INFO:teuthology.orchestra.run.smithi019.stdout: 5 251416 10.44 MB/sec execute 455 sec latency 55.862 ms 2024-03-26T02:03:02.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:02 smithi069 ceph-mon[53087]: pgmap v182: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 338 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.0 MiB/s wr, 407 op/s; 220/1662 objects degraded (13.237%) 2024-03-26T02:03:02.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:02 smithi069 ceph-mon[53087]: from='client.44259 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:03:02.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:02 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/2673332040' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:03:03.116 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:03:03 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3[80354]: 2024-03-26T02:03:03.094+0000 7f3b80169700 -1 osd.3 115 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:03:03.116 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:03 smithi019 ceph-mon[66359]: from='osd.3 [v2:172.21.15.19:6826/1307799479,v1:172.21.15.19:6827/1307799479]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T02:03:03.116 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:03 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:03.116 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:03 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:03.124 INFO:teuthology.orchestra.run.smithi019.stdout: 5 251702 10.44 MB/sec execute 456 sec latency 56.417 ms 2024-03-26T02:03:03.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:03 smithi069 ceph-mon[53087]: from='osd.3 [v2:172.21.15.19:6826/1307799479,v1:172.21.15.19:6827/1307799479]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T02:03:03.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:03 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:03.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:03 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:04.124 INFO:teuthology.orchestra.run.smithi019.stdout: 5 251960 10.42 MB/sec execute 457 sec latency 89.698 ms 2024-03-26T02:03:04.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:04 smithi019 ceph-mon[66359]: pgmap v183: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 338 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.5 MiB/s wr, 557 op/s; 208/1599 objects degraded (13.008%) 2024-03-26T02:03:04.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:04 smithi019 ceph-mon[66359]: from='osd.3 [v2:172.21.15.19:6826/1307799479,v1:172.21.15.19:6827/1307799479]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T02:03:04.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:04 smithi019 ceph-mon[66359]: osdmap e118: 8 total, 7 up, 8 in 2024-03-26T02:03:04.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:04 smithi019 ceph-mon[66359]: from='osd.3 [v2:172.21.15.19:6826/1307799479,v1:172.21.15.19:6827/1307799479]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-03-26T02:03:04.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:04 smithi019 ceph-mon[66359]: failed to encode map e118 with expected crc 2024-03-26T02:03:04.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:04 smithi019 ceph-mon[66359]: failed to encode map e118 with expected crc 2024-03-26T02:03:04.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:04 smithi019 ceph-mon[66359]: failed to encode map e118 with expected crc 2024-03-26T02:03:04.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:04 smithi069 ceph-mon[53087]: pgmap v183: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 338 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.5 MiB/s wr, 557 op/s; 208/1599 objects degraded (13.008%) 2024-03-26T02:03:04.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:04 smithi069 ceph-mon[53087]: from='osd.3 [v2:172.21.15.19:6826/1307799479,v1:172.21.15.19:6827/1307799479]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T02:03:04.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:04 smithi069 ceph-mon[53087]: osdmap e118: 8 total, 7 up, 8 in 2024-03-26T02:03:04.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:04 smithi069 ceph-mon[53087]: from='osd.3 [v2:172.21.15.19:6826/1307799479,v1:172.21.15.19:6827/1307799479]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-03-26T02:03:04.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:04 smithi069 ceph-mon[53087]: failed to encode map e118 with expected crc 2024-03-26T02:03:04.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:04 smithi069 ceph-mon[53087]: failed to encode map e118 with expected crc 2024-03-26T02:03:04.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:04 smithi069 ceph-mon[53087]: failed to encode map e118 with expected crc 2024-03-26T02:03:05.125 INFO:teuthology.orchestra.run.smithi019.stdout: 5 251960 10.40 MB/sec execute 458 sec latency 1089.740 ms 2024-03-26T02:03:05.158 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:05 smithi019 ceph-mon[66359]: failed to encode map e118 with expected crc 2024-03-26T02:03:05.158 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:05 smithi019 ceph-mon[66359]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:03:05.158 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:05 smithi019 ceph-mon[66359]: failed to encode map e119 with expected crc 2024-03-26T02:03:05.158 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:05 smithi019 ceph-mon[66359]: osd.3 [v2:172.21.15.19:6826/1307799479,v1:172.21.15.19:6827/1307799479] boot 2024-03-26T02:03:05.158 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:05 smithi019 ceph-mon[66359]: osdmap e119: 8 total, 8 up, 8 in 2024-03-26T02:03:05.158 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:03:05.158 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:05 smithi019 ceph-mon[66359]: failed to encode map e119 with expected crc 2024-03-26T02:03:05.158 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:05 smithi019 ceph-mon[66359]: failed to encode map e119 with expected crc 2024-03-26T02:03:05.158 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:05.158 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:05.159 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:05 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 208/1599 objects degraded (13.008%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T02:03:05.159 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:05 smithi019 ceph-mon[66359]: osdmap e120: 8 total, 8 up, 8 in 2024-03-26T02:03:05.159 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:05 smithi019 ceph-mon[66359]: failed to encode map e120 with expected crc 2024-03-26T02:03:05.159 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:05 smithi019 ceph-mon[66359]: failed to encode map e120 with expected crc 2024-03-26T02:03:05.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:05 smithi069 ceph-mon[53087]: failed to encode map e118 with expected crc 2024-03-26T02:03:05.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:05 smithi069 ceph-mon[53087]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:03:05.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:05 smithi069 ceph-mon[53087]: failed to encode map e119 with expected crc 2024-03-26T02:03:05.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:05 smithi069 ceph-mon[53087]: osd.3 [v2:172.21.15.19:6826/1307799479,v1:172.21.15.19:6827/1307799479] boot 2024-03-26T02:03:05.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:05 smithi069 ceph-mon[53087]: osdmap e119: 8 total, 8 up, 8 in 2024-03-26T02:03:05.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:03:05.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:05 smithi069 ceph-mon[53087]: failed to encode map e119 with expected crc 2024-03-26T02:03:05.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:05 smithi069 ceph-mon[53087]: failed to encode map e119 with expected crc 2024-03-26T02:03:05.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:05.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:05.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:05 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 208/1599 objects degraded (13.008%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T02:03:05.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:05 smithi069 ceph-mon[53087]: osdmap e120: 8 total, 8 up, 8 in 2024-03-26T02:03:05.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:05 smithi069 ceph-mon[53087]: failed to encode map e120 with expected crc 2024-03-26T02:03:05.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:05 smithi069 ceph-mon[53087]: failed to encode map e120 with expected crc 2024-03-26T02:03:06.125 INFO:teuthology.orchestra.run.smithi019.stdout: 5 251968 10.38 MB/sec execute 459 sec latency 2045.173 ms 2024-03-26T02:03:06.170 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:06 smithi019 ceph-mon[66359]: failed to encode map e120 with expected crc 2024-03-26T02:03:06.239 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:06 smithi019 ceph-mon[66359]: failed to encode map e120 with expected crc 2024-03-26T02:03:06.240 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:06 smithi019 ceph-mon[66359]: failed to encode map e120 with expected crc 2024-03-26T02:03:06.240 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:06 smithi019 ceph-mon[66359]: pgmap v187: 97 pgs: 13 active+undersized, 23 active+undersized+degraded, 61 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.1 MiB/s wr, 347 op/s; 187/1602 objects degraded (11.673%); 23 KiB/s, 2 objects/s recovering 2024-03-26T02:03:06.240 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:06 smithi019 ceph-mon[66359]: failed to encode map e121 with expected crc 2024-03-26T02:03:06.240 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:06 smithi019 ceph-mon[66359]: osdmap e121: 8 total, 8 up, 8 in 2024-03-26T02:03:06.240 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:06 smithi019 ceph-mon[66359]: failed to encode map e121 with expected crc 2024-03-26T02:03:06.241 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:06 smithi019 ceph-mon[66359]: failed to encode map e121 with expected crc 2024-03-26T02:03:06.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:06 smithi069 ceph-mon[53087]: failed to encode map e120 with expected crc 2024-03-26T02:03:06.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:06 smithi069 ceph-mon[53087]: failed to encode map e120 with expected crc 2024-03-26T02:03:06.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:06 smithi069 ceph-mon[53087]: failed to encode map e120 with expected crc 2024-03-26T02:03:06.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:06 smithi069 ceph-mon[53087]: pgmap v187: 97 pgs: 13 active+undersized, 23 active+undersized+degraded, 61 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.1 MiB/s wr, 347 op/s; 187/1602 objects degraded (11.673%); 23 KiB/s, 2 objects/s recovering 2024-03-26T02:03:06.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:06 smithi069 ceph-mon[53087]: failed to encode map e121 with expected crc 2024-03-26T02:03:06.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:06 smithi069 ceph-mon[53087]: osdmap e121: 8 total, 8 up, 8 in 2024-03-26T02:03:06.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:06 smithi069 ceph-mon[53087]: failed to encode map e121 with expected crc 2024-03-26T02:03:06.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:06 smithi069 ceph-mon[53087]: failed to encode map e121 with expected crc 2024-03-26T02:03:07.125 INFO:teuthology.orchestra.run.smithi019.stdout: 5 252203 10.36 MB/sec execute 460 sec latency 79.896 ms 2024-03-26T02:03:07.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:07 smithi019 ceph-mon[66359]: failed to encode map e121 with expected crc 2024-03-26T02:03:07.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:07 smithi019 ceph-mon[66359]: failed to encode map e121 with expected crc 2024-03-26T02:03:07.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:07 smithi069 ceph-mon[53087]: failed to encode map e121 with expected crc 2024-03-26T02:03:07.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:07 smithi069 ceph-mon[53087]: failed to encode map e121 with expected crc 2024-03-26T02:03:08.125 INFO:teuthology.orchestra.run.smithi019.stdout: 5 252210 10.34 MB/sec execute 461 sec latency 1038.020 ms 2024-03-26T02:03:08.221 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:08 smithi019 ceph-mon[66359]: pgmap v189: 97 pgs: 2 activating+undersized+degraded+remapped, 6 active+undersized, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 15 active+undersized+degraded, 69 active+clean; 336 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 2.0 MiB/s wr, 289 op/s; 185/1536 objects degraded (12.044%); 1.0 MiB/s, 5 objects/s recovering 2024-03-26T02:03:08.221 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:08 smithi019 ceph-mon[66359]: osdmap e122: 8 total, 8 up, 8 in 2024-03-26T02:03:08.221 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:08 smithi019 ceph-mon[66359]: failed to encode map e122 with expected crc 2024-03-26T02:03:08.221 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:08 smithi019 ceph-mon[66359]: failed to encode map e122 with expected crc 2024-03-26T02:03:08.221 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:08 smithi019 ceph-mon[66359]: failed to encode map e122 with expected crc 2024-03-26T02:03:08.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:08 smithi069 ceph-mon[53087]: pgmap v189: 97 pgs: 2 activating+undersized+degraded+remapped, 6 active+undersized, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 15 active+undersized+degraded, 69 active+clean; 336 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 2.0 MiB/s wr, 289 op/s; 185/1536 objects degraded (12.044%); 1.0 MiB/s, 5 objects/s recovering 2024-03-26T02:03:08.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:08 smithi069 ceph-mon[53087]: osdmap e122: 8 total, 8 up, 8 in 2024-03-26T02:03:08.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:08 smithi069 ceph-mon[53087]: failed to encode map e122 with expected crc 2024-03-26T02:03:08.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:08 smithi069 ceph-mon[53087]: failed to encode map e122 with expected crc 2024-03-26T02:03:08.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:08 smithi069 ceph-mon[53087]: failed to encode map e122 with expected crc 2024-03-26T02:03:09.125 INFO:teuthology.orchestra.run.smithi019.stdout: 5 252392 10.33 MB/sec execute 462 sec latency 905.390 ms 2024-03-26T02:03:09.280 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:09 smithi019 ceph-mon[66359]: failed to encode map e123 with expected crc 2024-03-26T02:03:09.280 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:09 smithi019 ceph-mon[66359]: failed to encode map e123 with expected crc 2024-03-26T02:03:09.280 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:09 smithi019 ceph-mon[66359]: osdmap e123: 8 total, 8 up, 8 in 2024-03-26T02:03:09.280 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:09 smithi019 ceph-mon[66359]: failed to encode map e123 with expected crc 2024-03-26T02:03:09.280 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:09 smithi019 ceph-mon[66359]: failed to encode map e123 with expected crc 2024-03-26T02:03:09.280 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:09 smithi019 ceph-mon[66359]: failed to encode map e123 with expected crc 2024-03-26T02:03:09.280 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:09 smithi019 ceph-mon[66359]: failed to encode map e123 with expected crc 2024-03-26T02:03:09.280 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:09 smithi019 ceph-mon[66359]: failed to encode map e123 with expected crc 2024-03-26T02:03:09.280 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:09 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:09.281 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:09 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:09.281 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:09 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:03:09.281 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:09 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:03:09.281 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:09 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:09.281 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:09 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:09.281 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:09 smithi019 ceph-mon[66359]: osdmap e124: 8 total, 8 up, 8 in 2024-03-26T02:03:09.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:09 smithi069 ceph-mon[53087]: failed to encode map e123 with expected crc 2024-03-26T02:03:09.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:09 smithi069 ceph-mon[53087]: failed to encode map e123 with expected crc 2024-03-26T02:03:09.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:09 smithi069 ceph-mon[53087]: osdmap e123: 8 total, 8 up, 8 in 2024-03-26T02:03:09.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:09 smithi069 ceph-mon[53087]: failed to encode map e123 with expected crc 2024-03-26T02:03:09.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:09 smithi069 ceph-mon[53087]: failed to encode map e123 with expected crc 2024-03-26T02:03:09.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:09 smithi069 ceph-mon[53087]: failed to encode map e123 with expected crc 2024-03-26T02:03:09.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:09 smithi069 ceph-mon[53087]: failed to encode map e123 with expected crc 2024-03-26T02:03:09.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:09 smithi069 ceph-mon[53087]: failed to encode map e123 with expected crc 2024-03-26T02:03:09.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:09 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:09.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:09 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:09.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:09 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:03:09.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:09 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:03:09.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:09 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:09.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:09 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:09.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:09 smithi069 ceph-mon[53087]: osdmap e124: 8 total, 8 up, 8 in 2024-03-26T02:03:10.125 INFO:teuthology.orchestra.run.smithi019.stdout: 5 252701 10.34 MB/sec execute 463 sec latency 225.139 ms 2024-03-26T02:03:10.288 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:10 smithi019 ceph-mon[66359]: pgmap v192: 97 pgs: 2 activating+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 84 active+clean; 336 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 2.8 MiB/s wr, 313 op/s; 211/1506 objects degraded (14.011%); 1017 KiB/s, 13 objects/s recovering 2024-03-26T02:03:10.288 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:10 smithi019 ceph-mon[66359]: failed to encode map e124 with expected crc 2024-03-26T02:03:10.288 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:10 smithi019 ceph-mon[66359]: failed to encode map e124 with expected crc 2024-03-26T02:03:10.288 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:10 smithi019 ceph-mon[66359]: failed to encode map e124 with expected crc 2024-03-26T02:03:10.288 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:10 smithi019 ceph-mon[66359]: failed to encode map e124 with expected crc 2024-03-26T02:03:10.288 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:10 smithi019 ceph-mon[66359]: failed to encode map e124 with expected crc 2024-03-26T02:03:10.288 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:10 smithi019 ceph-mon[66359]: failed to encode map e124 with expected crc 2024-03-26T02:03:10.288 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:10 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:03:10.288 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:10 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:03:10.288 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:10 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:03:10.289 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:10 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:03:10.289 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:10 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T02:03:10.289 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:10 smithi019 ceph-mon[66359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T02:03:10.289 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:10 smithi019 ceph-mon[66359]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-03-26T02:03:10.289 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:10 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 211/1506 objects degraded (14.011%), 12 pgs degraded (PG_DEGRADED) 2024-03-26T02:03:10.289 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:10 smithi019 ceph-mon[66359]: osdmap e125: 8 total, 8 up, 8 in 2024-03-26T02:03:10.289 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:10 smithi019 ceph-mon[66359]: failed to encode map e125 with expected crc 2024-03-26T02:03:10.289 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:10 smithi019 ceph-mon[66359]: failed to encode map e125 with expected crc 2024-03-26T02:03:10.289 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:10 smithi019 ceph-mon[66359]: failed to encode map e125 with expected crc 2024-03-26T02:03:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:10 smithi069 ceph-mon[53087]: pgmap v192: 97 pgs: 2 activating+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 84 active+clean; 336 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 2.8 MiB/s wr, 313 op/s; 211/1506 objects degraded (14.011%); 1017 KiB/s, 13 objects/s recovering 2024-03-26T02:03:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:10 smithi069 ceph-mon[53087]: failed to encode map e124 with expected crc 2024-03-26T02:03:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:10 smithi069 ceph-mon[53087]: failed to encode map e124 with expected crc 2024-03-26T02:03:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:10 smithi069 ceph-mon[53087]: failed to encode map e124 with expected crc 2024-03-26T02:03:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:10 smithi069 ceph-mon[53087]: failed to encode map e124 with expected crc 2024-03-26T02:03:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:10 smithi069 ceph-mon[53087]: failed to encode map e124 with expected crc 2024-03-26T02:03:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:10 smithi069 ceph-mon[53087]: failed to encode map e124 with expected crc 2024-03-26T02:03:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:10 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:03:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:10 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:03:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:10 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:03:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:10 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:03:10.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:10 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T02:03:10.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:10 smithi069 ceph-mon[53087]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T02:03:10.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:10 smithi069 ceph-mon[53087]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-03-26T02:03:10.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:10 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 211/1506 objects degraded (14.011%), 12 pgs degraded (PG_DEGRADED) 2024-03-26T02:03:10.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:10 smithi069 ceph-mon[53087]: osdmap e125: 8 total, 8 up, 8 in 2024-03-26T02:03:10.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:10 smithi069 ceph-mon[53087]: failed to encode map e125 with expected crc 2024-03-26T02:03:10.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:10 smithi069 ceph-mon[53087]: failed to encode map e125 with expected crc 2024-03-26T02:03:10.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:10 smithi069 ceph-mon[53087]: failed to encode map e125 with expected crc 2024-03-26T02:03:11.125 INFO:teuthology.orchestra.run.smithi019.stdout: 5 252974 10.33 MB/sec execute 464 sec latency 491.236 ms 2024-03-26T02:03:11.312 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:11 smithi019 ceph-mon[66359]: failed to encode map e125 with expected crc 2024-03-26T02:03:11.312 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:11 smithi019 ceph-mon[66359]: failed to encode map e125 with expected crc 2024-03-26T02:03:11.312 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:11 smithi019 ceph-mon[66359]: osdmap e126: 8 total, 8 up, 8 in 2024-03-26T02:03:11.313 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:11 smithi019 ceph-mon[66359]: failed to encode map e126 with expected crc 2024-03-26T02:03:11.313 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:11 smithi019 ceph-mon[66359]: failed to encode map e126 with expected crc 2024-03-26T02:03:11.313 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:11 smithi019 ceph-mon[66359]: pgmap v196: 97 pgs: 2 activating+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 84 active+clean; 334 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 558 B/s rd, 140 KiB/s wr, 19 op/s; 211/1488 objects degraded (14.180%); 54 KiB/s, 8 objects/s recovering 2024-03-26T02:03:11.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:11 smithi069 ceph-mon[53087]: failed to encode map e125 with expected crc 2024-03-26T02:03:11.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:11 smithi069 ceph-mon[53087]: failed to encode map e125 with expected crc 2024-03-26T02:03:11.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:11 smithi069 ceph-mon[53087]: osdmap e126: 8 total, 8 up, 8 in 2024-03-26T02:03:11.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:11 smithi069 ceph-mon[53087]: failed to encode map e126 with expected crc 2024-03-26T02:03:11.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:11 smithi069 ceph-mon[53087]: failed to encode map e126 with expected crc 2024-03-26T02:03:11.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:11 smithi069 ceph-mon[53087]: pgmap v196: 97 pgs: 2 activating+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 84 active+clean; 334 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 558 B/s rd, 140 KiB/s wr, 19 op/s; 211/1488 objects degraded (14.180%); 54 KiB/s, 8 objects/s recovering 2024-03-26T02:03:12.125 INFO:teuthology.orchestra.run.smithi019.stdout: 5 253345 10.32 MB/sec execute 465 sec latency 647.437 ms 2024-03-26T02:03:12.323 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:12 smithi019 ceph-mon[66359]: failed to encode map e126 with expected crc 2024-03-26T02:03:12.323 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:12 smithi019 ceph-mon[66359]: failed to encode map e127 with expected crc 2024-03-26T02:03:12.323 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:12 smithi019 ceph-mon[66359]: osdmap e127: 8 total, 8 up, 8 in 2024-03-26T02:03:12.323 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:12 smithi019 ceph-mon[66359]: failed to encode map e127 with expected crc 2024-03-26T02:03:12.323 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:12 smithi019 ceph-mon[66359]: failed to encode map e127 with expected crc 2024-03-26T02:03:12.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:12 smithi069 ceph-mon[53087]: failed to encode map e126 with expected crc 2024-03-26T02:03:12.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:12 smithi069 ceph-mon[53087]: failed to encode map e127 with expected crc 2024-03-26T02:03:12.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:12 smithi069 ceph-mon[53087]: osdmap e127: 8 total, 8 up, 8 in 2024-03-26T02:03:12.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:12 smithi069 ceph-mon[53087]: failed to encode map e127 with expected crc 2024-03-26T02:03:12.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:12 smithi069 ceph-mon[53087]: failed to encode map e127 with expected crc 2024-03-26T02:03:13.125 INFO:teuthology.orchestra.run.smithi019.stdout: 5 253617 10.31 MB/sec execute 466 sec latency 924.590 ms 2024-03-26T02:03:13.339 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:13 smithi019 ceph-mon[66359]: failed to encode map e127 with expected crc 2024-03-26T02:03:13.340 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:13 smithi019 ceph-mon[66359]: failed to encode map e128 with expected crc 2024-03-26T02:03:13.340 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:13 smithi019 ceph-mon[66359]: osdmap e128: 8 total, 8 up, 8 in 2024-03-26T02:03:13.340 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:13 smithi019 ceph-mon[66359]: pgmap v199: 97 pgs: 2 peering, 4 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 334 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 2.8 MiB/s wr, 379 op/s; 92/1596 objects degraded (5.764%); 3.1 MiB/s, 35 objects/s recovering 2024-03-26T02:03:13.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:13 smithi069 ceph-mon[53087]: failed to encode map e127 with expected crc 2024-03-26T02:03:13.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:13 smithi069 ceph-mon[53087]: failed to encode map e128 with expected crc 2024-03-26T02:03:13.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:13 smithi069 ceph-mon[53087]: osdmap e128: 8 total, 8 up, 8 in 2024-03-26T02:03:13.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:13 smithi069 ceph-mon[53087]: pgmap v199: 97 pgs: 2 peering, 4 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 334 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 2.8 MiB/s wr, 379 op/s; 92/1596 objects degraded (5.764%); 3.1 MiB/s, 35 objects/s recovering 2024-03-26T02:03:14.099 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:14 smithi019 ceph-mon[66359]: failed to encode map e128 with expected crc 2024-03-26T02:03:14.099 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:14 smithi019 ceph-mon[66359]: failed to encode map e128 with expected crc 2024-03-26T02:03:14.125 INFO:teuthology.orchestra.run.smithi019.stdout: 5 254123 10.33 MB/sec execute 467 sec latency 37.998 ms 2024-03-26T02:03:14.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:14 smithi069 ceph-mon[53087]: failed to encode map e128 with expected crc 2024-03-26T02:03:14.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:14 smithi069 ceph-mon[53087]: failed to encode map e128 with expected crc 2024-03-26T02:03:15.106 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:15 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 92/1596 objects degraded (5.764%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T02:03:15.106 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:15 smithi019 ceph-mon[66359]: pgmap v200: 97 pgs: 2 peering, 95 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 2.8 MiB/s wr, 299 op/s; 2.2 MiB/s, 41 objects/s recovering 2024-03-26T02:03:15.125 INFO:teuthology.orchestra.run.smithi019.stdout: 5 254581 10.33 MB/sec execute 468 sec latency 65.963 ms 2024-03-26T02:03:15.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:15 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 92/1596 objects degraded (5.764%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T02:03:15.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:15 smithi069 ceph-mon[53087]: pgmap v200: 97 pgs: 2 peering, 95 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 2.8 MiB/s wr, 299 op/s; 2.2 MiB/s, 41 objects/s recovering 2024-03-26T02:03:16.125 INFO:teuthology.orchestra.run.smithi019.stdout: 5 254853 10.33 MB/sec execute 469 sec latency 39.454 ms 2024-03-26T02:03:16.126 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:16 smithi019 ceph-mon[66359]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 92/1596 objects degraded (5.764%), 4 pgs degraded) 2024-03-26T02:03:16.126 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:16 smithi019 ceph-mon[66359]: Cluster is now healthy 2024-03-26T02:03:16.126 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:16 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:16.126 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:16 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:03:16.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:16 smithi069 ceph-mon[53087]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 92/1596 objects degraded (5.764%), 4 pgs degraded) 2024-03-26T02:03:16.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:16 smithi069 ceph-mon[53087]: Cluster is now healthy 2024-03-26T02:03:16.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:16 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:16.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:16 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:03:17.125 INFO:teuthology.orchestra.run.smithi019.stdout: 5 255140 10.33 MB/sec execute 470 sec latency 49.242 ms 2024-03-26T02:03:17.145 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:17 smithi019 ceph-mon[66359]: pgmap v201: 97 pgs: 2 peering, 95 active+clean; 338 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.9 MiB/s wr, 527 op/s; 1.9 MiB/s, 34 objects/s recovering 2024-03-26T02:03:17.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:17 smithi069 ceph-mon[53087]: pgmap v201: 97 pgs: 2 peering, 95 active+clean; 338 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.9 MiB/s wr, 527 op/s; 1.9 MiB/s, 34 objects/s recovering 2024-03-26T02:03:18.125 INFO:teuthology.orchestra.run.smithi019.stdout: 5 255485 10.31 MB/sec execute 471 sec latency 64.933 ms 2024-03-26T02:03:19.125 INFO:teuthology.orchestra.run.smithi019.stdout: 5 255758 10.31 MB/sec execute 472 sec latency 38.596 ms 2024-03-26T02:03:19.957 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:19 smithi019 ceph-mon[66359]: pgmap v202: 97 pgs: 97 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 5.2 MiB/s wr, 461 op/s; 1.5 MiB/s, 27 objects/s recovering 2024-03-26T02:03:20.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:19 smithi069 ceph-mon[53087]: pgmap v202: 97 pgs: 97 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 5.2 MiB/s wr, 461 op/s; 1.5 MiB/s, 27 objects/s recovering 2024-03-26T02:03:20.125 INFO:teuthology.orchestra.run.smithi019.stdout: 5 255960 10.30 MB/sec execute 473 sec latency 45.350 ms 2024-03-26T02:03:21.125 INFO:teuthology.orchestra.run.smithi019.stdout: 5 256254 10.30 MB/sec execute 474 sec latency 52.250 ms 2024-03-26T02:03:22.005 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:21 smithi019 ceph-mon[66359]: pgmap v203: 97 pgs: 97 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.5 MiB/s wr, 406 op/s; 252 KiB/s, 23 objects/s recovering 2024-03-26T02:03:22.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:21 smithi069 ceph-mon[53087]: pgmap v203: 97 pgs: 97 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.5 MiB/s wr, 406 op/s; 252 KiB/s, 23 objects/s recovering 2024-03-26T02:03:22.125 INFO:teuthology.orchestra.run.smithi019.stdout: 5 256633 10.29 MB/sec execute 475 sec latency 63.526 ms 2024-03-26T02:03:23.125 INFO:teuthology.orchestra.run.smithi019.stdout: 5 256981 10.28 MB/sec execute 476 sec latency 52.263 ms 2024-03-26T02:03:24.046 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:23 smithi019 ceph-mon[66359]: pgmap v204: 97 pgs: 97 active+clean; 335 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.8 MiB/s wr, 476 op/s; 82 KiB/s, 8 objects/s recovering 2024-03-26T02:03:24.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:23 smithi069 ceph-mon[53087]: pgmap v204: 97 pgs: 97 active+clean; 335 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.8 MiB/s wr, 476 op/s; 82 KiB/s, 8 objects/s recovering 2024-03-26T02:03:24.125 INFO:teuthology.orchestra.run.smithi019.stdout: 5 257167 10.27 MB/sec execute 477 sec latency 57.629 ms 2024-03-26T02:03:24.817 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:24 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T02:03:25.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:24 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T02:03:25.125 INFO:teuthology.orchestra.run.smithi019.stdout: 5 257469 10.28 MB/sec execute 478 sec latency 77.559 ms 2024-03-26T02:03:25.826 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:25 smithi019 ceph-mon[66359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T02:03:25.826 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:25 smithi019 ceph-mon[66359]: Upgrade: osd.4 is safe to restart 2024-03-26T02:03:25.826 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:25 smithi019 ceph-mon[66359]: pgmap v205: 97 pgs: 97 active+clean; 335 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.6 MiB/s wr, 432 op/s; 71 KiB/s, 7 objects/s recovering 2024-03-26T02:03:26.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:25 smithi069 ceph-mon[53087]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T02:03:26.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:25 smithi069 ceph-mon[53087]: Upgrade: osd.4 is safe to restart 2024-03-26T02:03:26.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:25 smithi069 ceph-mon[53087]: pgmap v205: 97 pgs: 97 active+clean; 335 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.6 MiB/s wr, 432 op/s; 71 KiB/s, 7 objects/s recovering 2024-03-26T02:03:26.126 INFO:teuthology.orchestra.run.smithi019.stdout: 5 257900 10.28 MB/sec execute 479 sec latency 58.825 ms 2024-03-26T02:03:26.826 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:26 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:26.837 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:26 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:26.837 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:26 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T02:03:26.837 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:26 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:03:27.085 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:26 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:27.086 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:26 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:27.086 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:26 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T02:03:27.086 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:26 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:03:27.126 INFO:teuthology.orchestra.run.smithi019.stdout: 5 258343 10.29 MB/sec execute 480 sec latency 40.116 ms 2024-03-26T02:03:28.082 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:27 smithi019 ceph-mon[66359]: Upgrade: Updating osd.4 2024-03-26T02:03:28.082 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:27 smithi019 ceph-mon[66359]: Deploying daemon osd.4 on smithi069 2024-03-26T02:03:28.082 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:27 smithi019 ceph-mon[66359]: pgmap v206: 97 pgs: 97 active+clean; 335 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.5 MiB/s wr, 558 op/s 2024-03-26T02:03:28.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:27 smithi069 ceph-mon[53087]: Upgrade: Updating osd.4 2024-03-26T02:03:28.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:27 smithi069 ceph-mon[53087]: Deploying daemon osd.4 on smithi069 2024-03-26T02:03:28.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:27 smithi069 ceph-mon[53087]: pgmap v206: 97 pgs: 97 active+clean; 335 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.5 MiB/s wr, 558 op/s 2024-03-26T02:03:28.125 INFO:teuthology.orchestra.run.smithi019.stdout: 5 258696 10.29 MB/sec execute 481 sec latency 37.625 ms 2024-03-26T02:03:29.126 INFO:teuthology.orchestra.run.smithi019.stdout: 5 259137 10.28 MB/sec execute 482 sec latency 35.714 ms 2024-03-26T02:03:29.874 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:29 smithi019 ceph-mon[66359]: pgmap v207: 97 pgs: 97 active+clean; 337 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.9 MiB/s wr, 478 op/s 2024-03-26T02:03:30.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:29 smithi069 ceph-mon[53087]: pgmap v207: 97 pgs: 97 active+clean; 337 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.9 MiB/s wr, 478 op/s 2024-03-26T02:03:30.110 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:30 smithi069 systemd[1]: Stopping Ceph osd.4 for 692607c6-eb12-11ee-95d0-87774f69a715... 2024-03-26T02:03:30.125 INFO:teuthology.orchestra.run.smithi019.stdout: 5 259459 10.28 MB/sec execute 483 sec latency 37.517 ms 2024-03-26T02:03:30.609 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:30 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4[31309]: 2024-03-26T02:03:30.347+0000 7f4a1d4d3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:03:30.610 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:30 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4[31309]: 2024-03-26T02:03:30.347+0000 7f4a1d4d3700 -1 osd.4 128 *** Got signal Terminated *** 2024-03-26T02:03:30.610 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:30 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4[31309]: 2024-03-26T02:03:30.347+0000 7f4a1d4d3700 -1 osd.4 128 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:03:30.954 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:30 smithi019 ceph-mon[66359]: osd.4 marked itself down and dead 2024-03-26T02:03:30.954 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:03:31.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:30 smithi069 ceph-mon[53087]: osd.4 marked itself down and dead 2024-03-26T02:03:31.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:03:31.126 INFO:teuthology.orchestra.run.smithi019.stdout: 5 259764 10.28 MB/sec execute 484 sec latency 291.470 ms 2024-03-26T02:03:31.798 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:31 smithi069 podman[57576]: 2024-03-26 02:03:31.491104038 +0000 UTC m=+1.277559544 container died 4fd5e853f8bfeba1806304abafcf43588bb07cad955262f7a84e0a12b6563136 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, io.openshift.tags=base centos centos-stream, architecture=x86_64, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., ceph=True, vcs-type=git, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, maintainer=Guillaume Abrioux ) 2024-03-26T02:03:31.995 INFO:teuthology.orchestra.run.smithi019.stdout:true 2024-03-26T02:03:32.107 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:31 smithi019 ceph-mon[66359]: pgmap v208: 97 pgs: 97 active+clean; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.2 MiB/s wr, 457 op/s 2024-03-26T02:03:32.108 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:31 smithi019 ceph-mon[66359]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:03:32.108 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:31 smithi019 ceph-mon[66359]: osdmap e129: 8 total, 7 up, 8 in 2024-03-26T02:03:32.108 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:31 smithi019 ceph-mon[66359]: failed to encode map e129 with expected crc 2024-03-26T02:03:32.108 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:31 smithi019 ceph-mon[66359]: failed to encode map e129 with expected crc 2024-03-26T02:03:32.108 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:31 smithi019 ceph-mon[66359]: failed to encode map e129 with expected crc 2024-03-26T02:03:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:31 smithi069 ceph-mon[53087]: pgmap v208: 97 pgs: 97 active+clean; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.2 MiB/s wr, 457 op/s 2024-03-26T02:03:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:31 smithi069 ceph-mon[53087]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:03:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:31 smithi069 ceph-mon[53087]: osdmap e129: 8 total, 7 up, 8 in 2024-03-26T02:03:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:31 smithi069 ceph-mon[53087]: failed to encode map e129 with expected crc 2024-03-26T02:03:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:31 smithi069 ceph-mon[53087]: failed to encode map e129 with expected crc 2024-03-26T02:03:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:31 smithi069 ceph-mon[53087]: failed to encode map e129 with expected crc 2024-03-26T02:03:32.109 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:31 smithi069 podman[57576]: 2024-03-26 02:03:31.88920419 +0000 UTC m=+1.675659691 container cleanup 4fd5e853f8bfeba1806304abafcf43588bb07cad955262f7a84e0a12b6563136 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4, name=centos-stream, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, ceph=True, RELEASE=HEAD, release=754, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, vendor=Red Hat, Inc., architecture=x86_64, io.k8s.display-name=CentOS Stream 8) 2024-03-26T02:03:32.110 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:31 smithi069 bash[57576]: ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4 2024-03-26T02:03:32.126 INFO:teuthology.orchestra.run.smithi019.stdout: 5 259809 10.26 MB/sec execute 485 sec latency 1044.861 ms 2024-03-26T02:03:32.581 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:32 smithi069 podman[57602]: 2024-03-26 02:03:32.199740653 +0000 UTC m=+0.710150477 container remove 4fd5e853f8bfeba1806304abafcf43588bb07cad955262f7a84e0a12b6563136 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4, ceph=True, io.openshift.expose-services=, version=8, architecture=x86_64, release=754, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, distribution-scope=public, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8) 2024-03-26T02:03:32.859 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:32 smithi069 podman[57681]: 2024-03-26 02:03:32.484922113 +0000 UTC m=+0.023330106 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:03:32.859 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:32 smithi069 podman[57681]: 2024-03-26 02:03:32.634105533 +0000 UTC m=+0.172513527 container create 4779fd88b4e946e8f68a2b4cc7d9961dd108d5b1aceb97a963541e2f0e49bace (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-26T02:03:32.886 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:32 smithi019 ceph-mon[66359]: failed to encode map e130 with expected crc 2024-03-26T02:03:32.886 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:32 smithi019 ceph-mon[66359]: osdmap e130: 8 total, 7 up, 8 in 2024-03-26T02:03:32.886 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:32 smithi019 ceph-mon[66359]: failed to encode map e130 with expected crc 2024-03-26T02:03:32.886 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:32 smithi019 ceph-mon[66359]: failed to encode map e130 with expected crc 2024-03-26T02:03:32.886 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:32 smithi019 ceph-mon[66359]: failed to encode map e130 with expected crc 2024-03-26T02:03:32.886 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:32 smithi019 ceph-mon[66359]: failed to encode map e130 with expected crc 2024-03-26T02:03:32.886 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:32 smithi019 ceph-mon[66359]: failed to encode map e130 with expected crc 2024-03-26T02:03:32.887 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:32 smithi019 ceph-mon[66359]: failed to encode map e130 with expected crc 2024-03-26T02:03:32.887 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:32 smithi019 ceph-mon[66359]: failed to encode map e130 with expected crc 2024-03-26T02:03:32.887 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:32 smithi019 ceph-mon[66359]: from='client.34346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:03:32.962 INFO:teuthology.orchestra.run.smithi019.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:03:32.962 INFO:teuthology.orchestra.run.smithi019.stdout:alertmanager.smithi019 smithi019 *:9093,9094 running (7m) 30s ago 16m 14.5M - 0.23.0 ba2b418f427c a6630841c2b2 2024-03-26T02:03:32.963 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi019 smithi019 running (3m) 30s ago 16m 6660k - 19.0.0-1775-g5e70124b 5f21c9264136 c5f91a47ddf2 2024-03-26T02:03:32.963 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi069 smithi069 running (3m) 3m ago 16m 6669k - 19.0.0-1775-g5e70124b 5f21c9264136 05128803b49e 2024-03-26T02:03:32.963 INFO:teuthology.orchestra.run.smithi019.stdout:grafana.smithi019 smithi019 *:3000 running (7m) 30s ago 16m 46.9M - 8.3.5 dad864ee21e9 a97dea3b5fec 2024-03-26T02:03:32.963 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi019.lmryom smithi019 running (10m) 30s ago 10m 38.4M - 17.2.0 e1d6a67b021e 91a7c2942715 2024-03-26T02:03:32.963 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi069.spgwgo smithi069 running (10m) 3m ago 10m 14.4M - 17.2.0 e1d6a67b021e 94d96338dbe1 2024-03-26T02:03:32.963 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi019.ixuvdm smithi019 *:8443,9283,8765 running (6m) 30s ago 17m 508M - 19.0.0-1775-g5e70124b 5f21c9264136 4460b22c7a56 2024-03-26T02:03:32.963 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi069.wpbewu smithi069 *:8443,9283,8765 running (5m) 3m ago 15m 451M - 19.0.0-1775-g5e70124b 5f21c9264136 380fd33842a8 2024-03-26T02:03:32.963 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi019 smithi019 running (5m) 30s ago 18m 52.0M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 c18934705774 2024-03-26T02:03:32.963 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi069 smithi069 running (4m) 3m ago 15m 35.6M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 a856bd263104 2024-03-26T02:03:32.963 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.0.0.smithi019.qbizxo smithi019 *:2049 running (10m) 30s ago 10m 283M - 4.0 e1d6a67b021e cd6628a9075d 2024-03-26T02:03:32.963 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.1.0.smithi069.uawdhc smithi069 *:2049 running (10m) 3m ago 10m 85.3M - 4.0 e1d6a67b021e 6ee3695e7a10 2024-03-26T02:03:32.963 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi019 smithi019 *:9100 running (7m) 30s ago 16m 10.2M - 1.3.1 1dbe0e931976 0d1d184bec63 2024-03-26T02:03:32.963 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi069 smithi069 *:9100 running (6m) 3m ago 15m 19.2M - 1.3.1 1dbe0e931976 c49a006f94e7 2024-03-26T02:03:32.963 INFO:teuthology.orchestra.run.smithi019.stdout:osd.0 smithi019 running (3m) 30s ago 15m 340M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 3f931cfe320a 2024-03-26T02:03:32.964 INFO:teuthology.orchestra.run.smithi019.stdout:osd.1 smithi019 running (2m) 30s ago 14m 289M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 758caf4ed841 2024-03-26T02:03:32.964 INFO:teuthology.orchestra.run.smithi019.stdout:osd.2 smithi019 running (91s) 30s ago 14m 228M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 6a9b701517ca 2024-03-26T02:03:32.964 INFO:teuthology.orchestra.run.smithi019.stdout:osd.3 smithi019 running (40s) 30s ago 13m 12.6M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 0401940d882b 2024-03-26T02:03:32.964 INFO:teuthology.orchestra.run.smithi019.stdout:osd.4 smithi069 running (13m) 3m ago 13m 556M 2704M 17.2.0 e1d6a67b021e 4fd5e853f8bf 2024-03-26T02:03:32.964 INFO:teuthology.orchestra.run.smithi019.stdout:osd.5 smithi069 running (12m) 3m ago 12m 433M 2704M 17.2.0 e1d6a67b021e 932e401d87c9 2024-03-26T02:03:32.964 INFO:teuthology.orchestra.run.smithi019.stdout:osd.6 smithi069 running (12m) 3m ago 12m 345M 2704M 17.2.0 e1d6a67b021e b123f75154c8 2024-03-26T02:03:32.964 INFO:teuthology.orchestra.run.smithi019.stdout:osd.7 smithi069 running (12m) 3m ago 12m 550M 2704M 17.2.0 e1d6a67b021e d175fbc9ff8a 2024-03-26T02:03:32.964 INFO:teuthology.orchestra.run.smithi019.stdout:prometheus.smithi019 smithi019 *:9095 running (5m) 30s ago 15m 67.6M - 2.33.4 514e6a882f6e 29f89db72a07 2024-03-26T02:03:33.126 INFO:teuthology.orchestra.run.smithi019.stdout: 5 260224 10.25 MB/sec execute 486 sec latency 57.822 ms 2024-03-26T02:03:33.243 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:32 smithi069 ceph-mon[53087]: failed to encode map e130 with expected crc 2024-03-26T02:03:33.243 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:32 smithi069 ceph-mon[53087]: osdmap e130: 8 total, 7 up, 8 in 2024-03-26T02:03:33.243 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:32 smithi069 ceph-mon[53087]: failed to encode map e130 with expected crc 2024-03-26T02:03:33.243 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:32 smithi069 ceph-mon[53087]: failed to encode map e130 with expected crc 2024-03-26T02:03:33.243 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:32 smithi069 ceph-mon[53087]: failed to encode map e130 with expected crc 2024-03-26T02:03:33.243 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:32 smithi069 ceph-mon[53087]: failed to encode map e130 with expected crc 2024-03-26T02:03:33.244 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:32 smithi069 ceph-mon[53087]: failed to encode map e130 with expected crc 2024-03-26T02:03:33.244 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:32 smithi069 ceph-mon[53087]: failed to encode map e130 with expected crc 2024-03-26T02:03:33.244 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:32 smithi069 ceph-mon[53087]: failed to encode map e130 with expected crc 2024-03-26T02:03:33.244 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:32 smithi069 ceph-mon[53087]: from='client.34346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:03:33.244 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:32 smithi069 podman[57681]: 2024-03-26 02:03:32.895684758 +0000 UTC m=+0.434092749 container init 4779fd88b4e946e8f68a2b4cc7d9961dd108d5b1aceb97a963541e2f0e49bace (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5) 2024-03-26T02:03:33.244 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:32 smithi069 podman[57681]: 2024-03-26 02:03:32.900193576 +0000 UTC m=+0.438601574 container start 4779fd88b4e946e8f68a2b4cc7d9961dd108d5b1aceb97a963541e2f0e49bace (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:03:33.244 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:32 smithi069 podman[57681]: 2024-03-26 02:03:32.995912674 +0000 UTC m=+0.534320685 container attach 4779fd88b4e946e8f68a2b4cc7d9961dd108d5b1aceb97a963541e2f0e49bace (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-26T02:03:33.543 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:03:33.543 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": { 2024-03-26T02:03:33.543 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:03:33.543 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:03:33.543 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": { 2024-03-26T02:03:33.543 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:03:33.543 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:03:33.543 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": { 2024-03-26T02:03:33.543 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-03-26T02:03:33.543 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 4 2024-03-26T02:03:33.544 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:03:33.544 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": { 2024-03-26T02:03:33.544 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:03:33.544 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:03:33.544 INFO:teuthology.orchestra.run.smithi019.stdout: "overall": { 2024-03-26T02:03:33.544 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-26T02:03:33.544 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 8 2024-03-26T02:03:33.544 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-26T02:03:33.544 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:03:33.609 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:33 smithi069 podman[57681]: 2024-03-26 02:03:33.243545814 +0000 UTC m=+0.781953813 container died 4779fd88b4e946e8f68a2b4cc7d9961dd108d5b1aceb97a963541e2f0e49bace (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0) 2024-03-26T02:03:33.923 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:33 smithi019 ceph-mon[66359]: from='client.44271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:03:33.924 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:33 smithi019 ceph-mon[66359]: pgmap v211: 97 pgs: 29 peering, 5 stale+active+clean, 63 active+clean; 342 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 5.7 MiB/s wr, 603 op/s 2024-03-26T02:03:33.924 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:33 smithi019 ceph-mon[66359]: Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-26T02:03:33.924 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:33 smithi019 ceph-mon[66359]: from='client.34354 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:03:33.924 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:33 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/2174808298' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:03:33.997 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:03:33.997 INFO:teuthology.orchestra.run.smithi019.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2", 2024-03-26T02:03:33.997 INFO:teuthology.orchestra.run.smithi019.stdout: "in_progress": true, 2024-03-26T02:03:33.997 INFO:teuthology.orchestra.run.smithi019.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T02:03:33.997 INFO:teuthology.orchestra.run.smithi019.stdout: "services_complete": [ 2024-03-26T02:03:33.997 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr", 2024-03-26T02:03:33.997 INFO:teuthology.orchestra.run.smithi019.stdout: "crash", 2024-03-26T02:03:33.997 INFO:teuthology.orchestra.run.smithi019.stdout: "mon" 2024-03-26T02:03:33.997 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-26T02:03:33.997 INFO:teuthology.orchestra.run.smithi019.stdout: "progress": "10/23 daemons upgraded", 2024-03-26T02:03:33.997 INFO:teuthology.orchestra.run.smithi019.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T02:03:33.998 INFO:teuthology.orchestra.run.smithi019.stdout: "is_paused": false 2024-03-26T02:03:33.998 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:03:34.081 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:33 smithi069 ceph-mon[53087]: from='client.44271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:03:34.081 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:33 smithi069 ceph-mon[53087]: pgmap v211: 97 pgs: 29 peering, 5 stale+active+clean, 63 active+clean; 342 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 5.7 MiB/s wr, 603 op/s 2024-03-26T02:03:34.081 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:33 smithi069 ceph-mon[53087]: Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-26T02:03:34.081 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:33 smithi069 ceph-mon[53087]: from='client.34354 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:03:34.081 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:33 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/2174808298' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:03:34.082 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:33 smithi069 podman[57707]: 2024-03-26 02:03:33.834216321 +0000 UTC m=+0.585362045 container remove 4779fd88b4e946e8f68a2b4cc7d9961dd108d5b1aceb97a963541e2f0e49bace (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux ) 2024-03-26T02:03:34.082 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:33 smithi069 systemd[1]: ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.4.service: Deactivated successfully. 2024-03-26T02:03:34.082 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:33 smithi069 systemd[1]: Stopped Ceph osd.4 for 692607c6-eb12-11ee-95d0-87774f69a715. 2024-03-26T02:03:34.082 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:33 smithi069 systemd[1]: ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.4.service: Consumed 4min 52.902s CPU time. 2024-03-26T02:03:34.082 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:34 smithi069 systemd[1]: Starting Ceph osd.4 for 692607c6-eb12-11ee-95d0-87774f69a715... 2024-03-26T02:03:34.126 INFO:teuthology.orchestra.run.smithi019.stdout: 5 260534 10.25 MB/sec execute 487 sec latency 74.474 ms 2024-03-26T02:03:34.359 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:34 smithi069 podman[57810]: 2024-03-26 02:03:34.224715982 +0000 UTC m=+0.021893986 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:03:34.561 INFO:teuthology.orchestra.run.smithi019.stdout:HEALTH_WARN 1 osds down; Reduced data availability: 4 pgs inactive, 5 pgs peering 2024-03-26T02:03:34.561 INFO:teuthology.orchestra.run.smithi019.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-26T02:03:34.561 INFO:teuthology.orchestra.run.smithi019.stdout: osd.4 (root=default,host=smithi069) is down 2024-03-26T02:03:34.561 INFO:teuthology.orchestra.run.smithi019.stdout:[WRN] PG_AVAILABILITY: Reduced data availability: 4 pgs inactive, 5 pgs peering 2024-03-26T02:03:34.561 INFO:teuthology.orchestra.run.smithi019.stdout: pg 4.1 is stuck peering for 10m, current state peering, last acting [5,6] 2024-03-26T02:03:34.562 INFO:teuthology.orchestra.run.smithi019.stdout: pg 4.d is stuck peering for 10m, current state peering, last acting [2,1] 2024-03-26T02:03:34.562 INFO:teuthology.orchestra.run.smithi019.stdout: pg 4.e is stuck peering for 2m, current state peering, last acting [6,1] 2024-03-26T02:03:34.562 INFO:teuthology.orchestra.run.smithi019.stdout: pg 4.13 is stuck peering for 2m, current state peering, last acting [6,1] 2024-03-26T02:03:34.562 INFO:teuthology.orchestra.run.smithi019.stdout: pg 4.18 is stuck peering for 81s, current state peering, last acting [6,2] 2024-03-26T02:03:34.805 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:34 smithi069 podman[57810]: 2024-03-26 02:03:34.460516223 +0000 UTC m=+0.257694224 container create 0ccd090419735d87d01e9c1eafa06a52cbb8731f6e125f7934e3a911b95f5b8d (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4-activate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-26T02:03:34.949 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:34 smithi019 ceph-mon[66359]: from='client.44283 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:03:34.949 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:34 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/2641020285' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:03:35.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:34 smithi069 ceph-mon[53087]: from='client.44283 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:03:35.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:34 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/2641020285' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:03:35.109 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:34 smithi069 podman[57810]: 2024-03-26 02:03:34.805821522 +0000 UTC m=+0.602999524 container init 0ccd090419735d87d01e9c1eafa06a52cbb8731f6e125f7934e3a911b95f5b8d (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4-activate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-26T02:03:35.109 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:34 smithi069 podman[57810]: 2024-03-26 02:03:34.810101874 +0000 UTC m=+0.607279885 container start 0ccd090419735d87d01e9c1eafa06a52cbb8731f6e125f7934e3a911b95f5b8d (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4-activate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True) 2024-03-26T02:03:35.110 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:34 smithi069 podman[57810]: 2024-03-26 02:03:34.913885179 +0000 UTC m=+0.711063204 container attach 0ccd090419735d87d01e9c1eafa06a52cbb8731f6e125f7934e3a911b95f5b8d (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4-activate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-26T02:03:35.126 INFO:teuthology.orchestra.run.smithi019.stdout: 5 260756 10.24 MB/sec execute 488 sec latency 41.516 ms 2024-03-26T02:03:35.609 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:35 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4-activate[57824]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-26T02:03:35.610 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:35 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4-activate[57824]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-26T02:03:35.610 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:35 smithi069 bash[57810]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-26T02:03:35.610 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:35 smithi069 bash[57810]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-26T02:03:35.610 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:35 smithi069 bash[57810]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-26T02:03:35.610 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:35 smithi069 bash[57810]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-26T02:03:35.610 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:35 smithi069 bash[57810]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-26T02:03:35.610 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:35 smithi069 bash[57810]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-26T02:03:35.610 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:35 smithi069 bash[57810]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-26T02:03:35.610 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:35 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4-activate[57824]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-26T02:03:35.610 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:35 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4-activate[57824]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-26T02:03:35.610 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:35 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4-activate[57824]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-26T02:03:35.610 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:35 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4-activate[57824]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-26T02:03:35.610 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:35 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4-activate[57824]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-26T02:03:35.611 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:35 smithi069 podman[57810]: 2024-03-26 02:03:35.355151046 +0000 UTC m=+1.152329040 container died 0ccd090419735d87d01e9c1eafa06a52cbb8731f6e125f7934e3a911b95f5b8d (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4-activate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-26T02:03:35.959 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:35 smithi019 ceph-mon[66359]: pgmap v212: 97 pgs: 7 active+undersized, 5 active+undersized+degraded, 29 peering, 56 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.5 MiB/s wr, 416 op/s; 53/1680 objects degraded (3.155%) 2024-03-26T02:03:35.959 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:35 smithi019 ceph-mon[66359]: Health check failed: Degraded data redundancy: 53/1680 objects degraded (3.155%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T02:03:35.968 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:35 smithi069 ceph-mon[53087]: pgmap v212: 97 pgs: 7 active+undersized, 5 active+undersized+degraded, 29 peering, 56 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.5 MiB/s wr, 416 op/s; 53/1680 objects degraded (3.155%) 2024-03-26T02:03:35.969 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:35 smithi069 ceph-mon[53087]: Health check failed: Degraded data redundancy: 53/1680 objects degraded (3.155%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T02:03:36.126 INFO:teuthology.orchestra.run.smithi019.stdout: 5 261055 10.24 MB/sec execute 489 sec latency 51.875 ms 2024-03-26T02:03:36.245 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:35 smithi069 podman[57957]: 2024-03-26 02:03:35.969043097 +0000 UTC m=+0.607902189 container remove 0ccd090419735d87d01e9c1eafa06a52cbb8731f6e125f7934e3a911b95f5b8d (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4-activate, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124) 2024-03-26T02:03:36.245 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:36 smithi069 podman[58016]: 2024-03-26 02:03:36.10032568 +0000 UTC m=+0.019785871 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:03:36.245 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:36 smithi069 podman[58016]: 2024-03-26 02:03:36.245220926 +0000 UTC m=+0.164681106 container create 6eb3b2ce99782f0593e80546272b83f504d3d49d8e09a8209a100a2545be3f11 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-26T02:03:36.859 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:36 smithi069 podman[58016]: 2024-03-26 02:03:36.523343752 +0000 UTC m=+0.442803943 container init 6eb3b2ce99782f0593e80546272b83f504d3d49d8e09a8209a100a2545be3f11 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319) 2024-03-26T02:03:36.859 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:36 smithi069 podman[58016]: 2024-03-26 02:03:36.527868057 +0000 UTC m=+0.447328240 container start 6eb3b2ce99782f0593e80546272b83f504d3d49d8e09a8209a100a2545be3f11 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:03:36.859 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:36 smithi069 ceph-osd[58034]: -- 172.21.15.69:0/3479064392 <== mon.1 v2:172.21.15.69:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5642a51131e0 con 0x5642a50e4c00 2024-03-26T02:03:36.859 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:36 smithi069 bash[58016]: 6eb3b2ce99782f0593e80546272b83f504d3d49d8e09a8209a100a2545be3f11 2024-03-26T02:03:36.859 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:36 smithi069 systemd[1]: Started Ceph osd.4 for 692607c6-eb12-11ee-95d0-87774f69a715. 2024-03-26T02:03:37.126 INFO:teuthology.orchestra.run.smithi019.stdout: 5 261390 10.24 MB/sec execute 490 sec latency 70.427 ms 2024-03-26T02:03:37.359 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:37 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4[58030]: 2024-03-26T02:03:37.068+0000 7fbab9cfa7c0 -1 Falling back to public interface 2024-03-26T02:03:37.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:37 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:37.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:37 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:37.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:37 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:03:37.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:37 smithi019 ceph-mon[66359]: pgmap v213: 97 pgs: 12 active+undersized, 21 active+undersized+degraded, 8 peering, 56 active+clean; 334 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.7 MiB/s wr, 531 op/s; 204/1638 objects degraded (12.454%) 2024-03-26T02:03:38.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:37 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:38.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:37 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:38.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:37 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:03:38.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:37 smithi069 ceph-mon[53087]: pgmap v213: 97 pgs: 12 active+undersized, 21 active+undersized+degraded, 8 peering, 56 active+clean; 334 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 4.7 MiB/s wr, 531 op/s; 204/1638 objects degraded (12.454%) 2024-03-26T02:03:38.126 INFO:teuthology.orchestra.run.smithi019.stdout: 5 261689 10.24 MB/sec execute 491 sec latency 63.193 ms 2024-03-26T02:03:38.831 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:38 smithi019 ceph-mon[66359]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-03-26T02:03:39.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:38 smithi069 ceph-mon[53087]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-03-26T02:03:39.126 INFO:teuthology.orchestra.run.smithi019.stdout: 5 261887 10.24 MB/sec execute 492 sec latency 67.145 ms 2024-03-26T02:03:39.838 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:39 smithi019 ceph-mon[66359]: pgmap v214: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 5.5 MiB/s wr, 566 op/s; 268/1656 objects degraded (16.184%) 2024-03-26T02:03:40.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:39 smithi069 ceph-mon[53087]: pgmap v214: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 5.5 MiB/s wr, 566 op/s; 268/1656 objects degraded (16.184%) 2024-03-26T02:03:40.126 INFO:teuthology.orchestra.run.smithi019.stdout: 5 262139 10.23 MB/sec execute 493 sec latency 46.543 ms 2024-03-26T02:03:41.126 INFO:teuthology.orchestra.run.smithi019.stdout: 5 262451 10.22 MB/sec execute 494 sec latency 40.012 ms 2024-03-26T02:03:41.870 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:41 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:41.871 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:41 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:41.871 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:41 smithi019 ceph-mon[66359]: pgmap v215: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.6 MiB/s wr, 466 op/s; 265/1638 objects degraded (16.178%) 2024-03-26T02:03:42.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:41 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:42.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:41 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:42.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:41 smithi069 ceph-mon[53087]: pgmap v215: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.6 MiB/s wr, 466 op/s; 265/1638 objects degraded (16.178%) 2024-03-26T02:03:42.126 INFO:teuthology.orchestra.run.smithi019.stdout: 5 262769 10.21 MB/sec execute 495 sec latency 47.479 ms 2024-03-26T02:03:42.890 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:42 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 265/1638 objects degraded (16.178%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T02:03:42.891 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:42 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:42.891 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:42 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:42.936 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:42 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 265/1638 objects degraded (16.178%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T02:03:42.936 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:42 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:42.936 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:42 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:43.126 INFO:teuthology.orchestra.run.smithi019.stdout: 5 262970 10.20 MB/sec execute 496 sec latency 59.856 ms 2024-03-26T02:03:43.916 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:43 smithi019 ceph-mon[66359]: pgmap v216: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 338 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.9 MiB/s wr, 387 op/s; 266/1686 objects degraded (15.777%) 2024-03-26T02:03:44.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:43 smithi069 ceph-mon[53087]: pgmap v216: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 338 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.9 MiB/s wr, 387 op/s; 266/1686 objects degraded (15.777%) 2024-03-26T02:03:44.126 INFO:teuthology.orchestra.run.smithi019.stdout: 5 263243 10.20 MB/sec execute 497 sec latency 70.438 ms 2024-03-26T02:03:45.099 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:44 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4[58030]: 2024-03-26T02:03:44.660+0000 7fbab9cfa7c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-03-26T02:03:45.126 INFO:teuthology.orchestra.run.smithi019.stdout: 5 263461 10.19 MB/sec execute 498 sec latency 89.504 ms 2024-03-26T02:03:45.974 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:45 smithi019 ceph-mon[66359]: pgmap v217: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 339 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.9 MiB/s wr, 367 op/s; 267/1692 objects degraded (15.780%) 2024-03-26T02:03:45.974 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:45.974 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:03:46.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:45 smithi069 ceph-mon[53087]: pgmap v217: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 339 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.9 MiB/s wr, 367 op/s; 267/1692 objects degraded (15.780%) 2024-03-26T02:03:46.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:46.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:03:46.126 INFO:teuthology.orchestra.run.smithi019.stdout: 5 263775 10.18 MB/sec execute 499 sec latency 79.046 ms 2024-03-26T02:03:46.609 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:46 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4[58030]: 2024-03-26T02:03:46.238+0000 7fbab9cfa7c0 -1 osd.4 128 log_to_monitors true 2024-03-26T02:03:47.013 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:46 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:47.013 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:46 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:47.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:46 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:03:47.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:46 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:03:47.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:46 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:47.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:46 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:47.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:46 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:03:47.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:46 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:03:47.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:46 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:03:47.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:46 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:03:47.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:46 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T02:03:47.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:46 smithi019 ceph-mon[66359]: from='osd.4 [v2:172.21.15.69:6800/1213736564,v1:172.21.15.69:6801/1213736564]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T02:03:47.014 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:46 smithi019 ceph-mon[66359]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T02:03:47.127 INFO:teuthology.orchestra.run.smithi019.stdout: 5 264097 10.17 MB/sec execute 500 sec latency 51.423 ms 2024-03-26T02:03:47.359 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:03:46 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4[58030]: 2024-03-26T02:03:46.987+0000 7fbaab152700 -1 osd.4 128 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:03:47.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:46 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:47.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:46 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:47.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:46 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:03:47.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:46 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:03:47.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:46 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:47.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:46 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:03:47.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:46 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:03:47.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:46 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:03:47.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:46 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:03:47.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:46 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:03:47.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:46 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T02:03:47.361 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:46 smithi069 ceph-mon[53087]: from='osd.4 [v2:172.21.15.69:6800/1213736564,v1:172.21.15.69:6801/1213736564]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T02:03:47.361 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:46 smithi069 ceph-mon[53087]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T02:03:48.033 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:47 smithi019 ceph-mon[66359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T02:03:48.033 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:47 smithi019 ceph-mon[66359]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-03-26T02:03:48.033 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:47 smithi019 ceph-mon[66359]: pgmap v218: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 338 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 4.4 MiB/s wr, 484 op/s; 252/1644 objects degraded (15.328%) 2024-03-26T02:03:48.034 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:47 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 252/1644 objects degraded (15.328%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T02:03:48.034 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:47 smithi019 ceph-mon[66359]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T02:03:48.034 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:47 smithi019 ceph-mon[66359]: failed to encode map e131 with expected crc 2024-03-26T02:03:48.034 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:47 smithi019 ceph-mon[66359]: osdmap e131: 8 total, 7 up, 8 in 2024-03-26T02:03:48.034 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:47 smithi019 ceph-mon[66359]: from='osd.4 [v2:172.21.15.69:6800/1213736564,v1:172.21.15.69:6801/1213736564]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-03-26T02:03:48.034 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:47 smithi019 ceph-mon[66359]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-03-26T02:03:48.126 INFO:teuthology.orchestra.run.smithi019.stdout: 5 264304 10.16 MB/sec execute 501 sec latency 152.463 ms 2024-03-26T02:03:48.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:47 smithi069 ceph-mon[53087]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T02:03:48.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:47 smithi069 ceph-mon[53087]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-03-26T02:03:48.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:47 smithi069 ceph-mon[53087]: pgmap v218: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 338 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 4.4 MiB/s wr, 484 op/s; 252/1644 objects degraded (15.328%) 2024-03-26T02:03:48.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:47 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 252/1644 objects degraded (15.328%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T02:03:48.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:47 smithi069 ceph-mon[53087]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T02:03:48.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:47 smithi069 ceph-mon[53087]: failed to encode map e131 with expected crc 2024-03-26T02:03:48.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:47 smithi069 ceph-mon[53087]: osdmap e131: 8 total, 7 up, 8 in 2024-03-26T02:03:48.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:47 smithi069 ceph-mon[53087]: from='osd.4 [v2:172.21.15.69:6800/1213736564,v1:172.21.15.69:6801/1213736564]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-03-26T02:03:48.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:47 smithi069 ceph-mon[53087]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-03-26T02:03:49.126 INFO:teuthology.orchestra.run.smithi019.stdout: 5 264323 10.14 MB/sec execute 502 sec latency 1152.497 ms 2024-03-26T02:03:49.175 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:48 smithi019 ceph-mon[66359]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:03:49.175 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:48 smithi019 ceph-mon[66359]: failed to encode map e132 with expected crc 2024-03-26T02:03:49.175 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:48 smithi019 ceph-mon[66359]: osd.4 [v2:172.21.15.69:6800/1213736564,v1:172.21.15.69:6801/1213736564] boot 2024-03-26T02:03:49.175 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:48 smithi019 ceph-mon[66359]: osdmap e132: 8 total, 8 up, 8 in 2024-03-26T02:03:49.175 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:48 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:03:49.175 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:48 smithi019 ceph-mon[66359]: failed to encode map e132 with expected crc 2024-03-26T02:03:49.175 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:48 smithi019 ceph-mon[66359]: failed to encode map e132 with expected crc 2024-03-26T02:03:49.206 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:48 smithi069 ceph-mon[53087]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:03:49.206 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:48 smithi069 ceph-mon[53087]: failed to encode map e132 with expected crc 2024-03-26T02:03:49.206 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:48 smithi069 ceph-mon[53087]: osd.4 [v2:172.21.15.69:6800/1213736564,v1:172.21.15.69:6801/1213736564] boot 2024-03-26T02:03:49.206 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:48 smithi069 ceph-mon[53087]: osdmap e132: 8 total, 8 up, 8 in 2024-03-26T02:03:49.206 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:48 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:03:49.206 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:48 smithi069 ceph-mon[53087]: failed to encode map e132 with expected crc 2024-03-26T02:03:49.206 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:48 smithi069 ceph-mon[53087]: failed to encode map e132 with expected crc 2024-03-26T02:03:50.126 INFO:teuthology.orchestra.run.smithi019.stdout: 5 264763 10.15 MB/sec execute 503 sec latency 2040.769 ms 2024-03-26T02:03:50.193 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:49 smithi019 ceph-mon[66359]: pgmap v221: 97 pgs: 2 peering, 12 active+undersized, 27 active+undersized+degraded, 56 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.5 MiB/s wr, 496 op/s; 230/1599 objects degraded (14.384%) 2024-03-26T02:03:50.193 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:49 smithi019 ceph-mon[66359]: failed to encode map e133 with expected crc 2024-03-26T02:03:50.193 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:49 smithi019 ceph-mon[66359]: osdmap e133: 8 total, 8 up, 8 in 2024-03-26T02:03:50.193 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:49 smithi019 ceph-mon[66359]: failed to encode map e133 with expected crc 2024-03-26T02:03:50.193 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:49 smithi019 ceph-mon[66359]: failed to encode map e133 with expected crc 2024-03-26T02:03:50.193 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:49 smithi019 ceph-mon[66359]: failed to encode map e133 with expected crc 2024-03-26T02:03:50.193 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:49 smithi019 ceph-mon[66359]: failed to encode map e133 with expected crc 2024-03-26T02:03:50.193 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:49 smithi019 ceph-mon[66359]: failed to encode map e133 with expected crc 2024-03-26T02:03:50.193 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:49 smithi019 ceph-mon[66359]: failed to encode map e133 with expected crc 2024-03-26T02:03:50.193 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:49 smithi019 ceph-mon[66359]: failed to encode map e133 with expected crc 2024-03-26T02:03:50.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:49 smithi069 ceph-mon[53087]: pgmap v221: 97 pgs: 2 peering, 12 active+undersized, 27 active+undersized+degraded, 56 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.5 MiB/s wr, 496 op/s; 230/1599 objects degraded (14.384%) 2024-03-26T02:03:50.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:49 smithi069 ceph-mon[53087]: failed to encode map e133 with expected crc 2024-03-26T02:03:50.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:49 smithi069 ceph-mon[53087]: osdmap e133: 8 total, 8 up, 8 in 2024-03-26T02:03:50.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:49 smithi069 ceph-mon[53087]: failed to encode map e133 with expected crc 2024-03-26T02:03:50.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:49 smithi069 ceph-mon[53087]: failed to encode map e133 with expected crc 2024-03-26T02:03:50.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:49 smithi069 ceph-mon[53087]: failed to encode map e133 with expected crc 2024-03-26T02:03:50.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:49 smithi069 ceph-mon[53087]: failed to encode map e133 with expected crc 2024-03-26T02:03:50.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:49 smithi069 ceph-mon[53087]: failed to encode map e133 with expected crc 2024-03-26T02:03:50.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:49 smithi069 ceph-mon[53087]: failed to encode map e133 with expected crc 2024-03-26T02:03:50.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:49 smithi069 ceph-mon[53087]: failed to encode map e133 with expected crc 2024-03-26T02:03:51.127 INFO:teuthology.orchestra.run.smithi019.stdout: 5 265187 10.16 MB/sec execute 504 sec latency 45.605 ms 2024-03-26T02:03:51.215 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:50 smithi019 ceph-mon[66359]: osdmap e134: 8 total, 8 up, 8 in 2024-03-26T02:03:51.215 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:50 smithi019 ceph-mon[66359]: failed to encode map e134 with expected crc 2024-03-26T02:03:51.215 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:50 smithi019 ceph-mon[66359]: failed to encode map e134 with expected crc 2024-03-26T02:03:51.215 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:50 smithi019 ceph-mon[66359]: failed to encode map e134 with expected crc 2024-03-26T02:03:51.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:50 smithi069 ceph-mon[53087]: osdmap e134: 8 total, 8 up, 8 in 2024-03-26T02:03:51.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:50 smithi069 ceph-mon[53087]: failed to encode map e134 with expected crc 2024-03-26T02:03:51.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:50 smithi069 ceph-mon[53087]: failed to encode map e134 with expected crc 2024-03-26T02:03:51.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:50 smithi069 ceph-mon[53087]: failed to encode map e134 with expected crc 2024-03-26T02:03:52.129 INFO:teuthology.orchestra.run.smithi019.stdout: 5 265542 10.17 MB/sec execute 505 sec latency 935.877 ms 2024-03-26T02:03:52.233 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:51 smithi019 ceph-mon[66359]: pgmap v224: 97 pgs: 1 remapped+peering, 2 peering, 10 active+undersized, 23 active+undersized+degraded, 61 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.8 MiB/s wr, 112 op/s; 199/1635 objects degraded (12.171%); 28 KiB/s, 50 keys/s, 4 objects/s recovering 2024-03-26T02:03:52.233 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:51 smithi019 ceph-mon[66359]: failed to encode map e135 with expected crc 2024-03-26T02:03:52.233 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:51 smithi019 ceph-mon[66359]: osdmap e135: 8 total, 8 up, 8 in 2024-03-26T02:03:52.233 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:51 smithi019 ceph-mon[66359]: failed to encode map e135 with expected crc 2024-03-26T02:03:52.233 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:51 smithi019 ceph-mon[66359]: failed to encode map e135 with expected crc 2024-03-26T02:03:52.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:51 smithi069 ceph-mon[53087]: pgmap v224: 97 pgs: 1 remapped+peering, 2 peering, 10 active+undersized, 23 active+undersized+degraded, 61 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.8 MiB/s wr, 112 op/s; 199/1635 objects degraded (12.171%); 28 KiB/s, 50 keys/s, 4 objects/s recovering 2024-03-26T02:03:52.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:51 smithi069 ceph-mon[53087]: failed to encode map e135 with expected crc 2024-03-26T02:03:52.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:51 smithi069 ceph-mon[53087]: osdmap e135: 8 total, 8 up, 8 in 2024-03-26T02:03:52.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:51 smithi069 ceph-mon[53087]: failed to encode map e135 with expected crc 2024-03-26T02:03:52.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:51 smithi069 ceph-mon[53087]: failed to encode map e135 with expected crc 2024-03-26T02:03:53.127 INFO:teuthology.orchestra.run.smithi019.stdout: 5 265757 10.16 MB/sec execute 506 sec latency 512.532 ms 2024-03-26T02:03:53.249 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:52 smithi019 ceph-mon[66359]: failed to encode map e136 with expected crc 2024-03-26T02:03:53.249 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:52 smithi019 ceph-mon[66359]: osdmap e136: 8 total, 8 up, 8 in 2024-03-26T02:03:53.249 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:52 smithi019 ceph-mon[66359]: failed to encode map e136 with expected crc 2024-03-26T02:03:53.249 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:52 smithi019 ceph-mon[66359]: failed to encode map e136 with expected crc 2024-03-26T02:03:53.249 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:52 smithi019 ceph-mon[66359]: failed to encode map e136 with expected crc 2024-03-26T02:03:53.249 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:52 smithi019 ceph-mon[66359]: failed to encode map e136 with expected crc 2024-03-26T02:03:53.249 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:52 smithi019 ceph-mon[66359]: failed to encode map e136 with expected crc 2024-03-26T02:03:53.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:52 smithi069 ceph-mon[53087]: failed to encode map e136 with expected crc 2024-03-26T02:03:53.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:52 smithi069 ceph-mon[53087]: osdmap e136: 8 total, 8 up, 8 in 2024-03-26T02:03:53.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:52 smithi069 ceph-mon[53087]: failed to encode map e136 with expected crc 2024-03-26T02:03:53.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:52 smithi069 ceph-mon[53087]: failed to encode map e136 with expected crc 2024-03-26T02:03:53.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:52 smithi069 ceph-mon[53087]: failed to encode map e136 with expected crc 2024-03-26T02:03:53.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:52 smithi069 ceph-mon[53087]: failed to encode map e136 with expected crc 2024-03-26T02:03:53.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:52 smithi069 ceph-mon[53087]: failed to encode map e136 with expected crc 2024-03-26T02:03:54.013 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:53 smithi019 ceph-mon[66359]: pgmap v227: 97 pgs: 1 active+recovering+degraded, 2 active+recovery_wait+degraded, 2 peering, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 328 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.0 MiB/s wr, 241 op/s; 52/1692 objects degraded (3.073%); 586 KiB/s, 50 keys/s, 58 objects/s recovering 2024-03-26T02:03:54.013 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:53 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 52/1692 objects degraded (3.073%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T02:03:54.013 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:53 smithi019 ceph-mon[66359]: failed to encode map e137 with expected crc 2024-03-26T02:03:54.013 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:53 smithi019 ceph-mon[66359]: failed to encode map e137 with expected crc 2024-03-26T02:03:54.013 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:53 smithi019 ceph-mon[66359]: osdmap e137: 8 total, 8 up, 8 in 2024-03-26T02:03:54.013 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:53 smithi019 ceph-mon[66359]: failed to encode map e137 with expected crc 2024-03-26T02:03:54.126 INFO:teuthology.orchestra.run.smithi019.stdout: 5 266059 10.15 MB/sec execute 507 sec latency 50.619 ms 2024-03-26T02:03:54.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:53 smithi069 ceph-mon[53087]: pgmap v227: 97 pgs: 1 active+recovering+degraded, 2 active+recovery_wait+degraded, 2 peering, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 328 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.0 MiB/s wr, 241 op/s; 52/1692 objects degraded (3.073%); 586 KiB/s, 50 keys/s, 58 objects/s recovering 2024-03-26T02:03:54.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:53 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 52/1692 objects degraded (3.073%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T02:03:54.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:53 smithi069 ceph-mon[53087]: failed to encode map e137 with expected crc 2024-03-26T02:03:54.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:53 smithi069 ceph-mon[53087]: failed to encode map e137 with expected crc 2024-03-26T02:03:54.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:53 smithi069 ceph-mon[53087]: osdmap e137: 8 total, 8 up, 8 in 2024-03-26T02:03:54.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:53 smithi069 ceph-mon[53087]: failed to encode map e137 with expected crc 2024-03-26T02:03:55.048 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:55 smithi019 ceph-mon[66359]: failed to encode map e138 with expected crc 2024-03-26T02:03:55.048 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:55 smithi019 ceph-mon[66359]: osdmap e138: 8 total, 8 up, 8 in 2024-03-26T02:03:55.048 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:55 smithi019 ceph-mon[66359]: failed to encode map e138 with expected crc 2024-03-26T02:03:55.048 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:55 smithi019 ceph-mon[66359]: failed to encode map e138 with expected crc 2024-03-26T02:03:55.048 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:55 smithi019 ceph-mon[66359]: failed to encode map e138 with expected crc 2024-03-26T02:03:55.048 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:55 smithi019 ceph-mon[66359]: failed to encode map e138 with expected crc 2024-03-26T02:03:55.126 INFO:teuthology.orchestra.run.smithi019.stdout: 5 266322 10.14 MB/sec execute 508 sec latency 257.334 ms 2024-03-26T02:03:55.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:55 smithi069 ceph-mon[53087]: failed to encode map e138 with expected crc 2024-03-26T02:03:55.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:55 smithi069 ceph-mon[53087]: osdmap e138: 8 total, 8 up, 8 in 2024-03-26T02:03:55.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:55 smithi069 ceph-mon[53087]: failed to encode map e138 with expected crc 2024-03-26T02:03:55.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:55 smithi069 ceph-mon[53087]: failed to encode map e138 with expected crc 2024-03-26T02:03:55.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:55 smithi069 ceph-mon[53087]: failed to encode map e138 with expected crc 2024-03-26T02:03:55.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:55 smithi069 ceph-mon[53087]: failed to encode map e138 with expected crc 2024-03-26T02:03:56.065 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:56 smithi019 ceph-mon[66359]: pgmap v230: 97 pgs: 1 active+recovering+degraded, 2 active+recovery_wait+degraded, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 337 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 6.9 MiB/s wr, 728 op/s; 52/1728 objects degraded (3.009%); 642 KiB/s, 58 objects/s recovering 2024-03-26T02:03:56.066 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:56 smithi019 ceph-mon[66359]: failed to encode map e139 with expected crc 2024-03-26T02:03:56.066 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:56 smithi019 ceph-mon[66359]: failed to encode map e139 with expected crc 2024-03-26T02:03:56.066 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:56 smithi019 ceph-mon[66359]: osdmap e139: 8 total, 8 up, 8 in 2024-03-26T02:03:56.066 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:56 smithi019 ceph-mon[66359]: failed to encode map e139 with expected crc 2024-03-26T02:03:56.066 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:56 smithi019 ceph-mon[66359]: failed to encode map e139 with expected crc 2024-03-26T02:03:56.066 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:56 smithi019 ceph-mon[66359]: failed to encode map e139 with expected crc 2024-03-26T02:03:56.066 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:56 smithi019 ceph-mon[66359]: failed to encode map e139 with expected crc 2024-03-26T02:03:56.127 INFO:teuthology.orchestra.run.smithi019.stdout: 5 266626 10.13 MB/sec execute 509 sec latency 44.876 ms 2024-03-26T02:03:56.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:56 smithi069 ceph-mon[53087]: pgmap v230: 97 pgs: 1 active+recovering+degraded, 2 active+recovery_wait+degraded, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 337 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 6.9 MiB/s wr, 728 op/s; 52/1728 objects degraded (3.009%); 642 KiB/s, 58 objects/s recovering 2024-03-26T02:03:56.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:56 smithi069 ceph-mon[53087]: failed to encode map e139 with expected crc 2024-03-26T02:03:56.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:56 smithi069 ceph-mon[53087]: failed to encode map e139 with expected crc 2024-03-26T02:03:56.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:56 smithi069 ceph-mon[53087]: osdmap e139: 8 total, 8 up, 8 in 2024-03-26T02:03:56.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:56 smithi069 ceph-mon[53087]: failed to encode map e139 with expected crc 2024-03-26T02:03:56.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:56 smithi069 ceph-mon[53087]: failed to encode map e139 with expected crc 2024-03-26T02:03:56.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:56 smithi069 ceph-mon[53087]: failed to encode map e139 with expected crc 2024-03-26T02:03:56.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:56 smithi069 ceph-mon[53087]: failed to encode map e139 with expected crc 2024-03-26T02:03:57.108 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:57 smithi019 ceph-mon[66359]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 52/1728 objects degraded (3.009%), 4 pgs degraded) 2024-03-26T02:03:57.109 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:57 smithi019 ceph-mon[66359]: Cluster is now healthy 2024-03-26T02:03:57.127 INFO:teuthology.orchestra.run.smithi019.stdout: 5 266952 10.13 MB/sec execute 510 sec latency 68.759 ms 2024-03-26T02:03:57.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:57 smithi069 ceph-mon[53087]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 52/1728 objects degraded (3.009%), 4 pgs degraded) 2024-03-26T02:03:57.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:57 smithi069 ceph-mon[53087]: Cluster is now healthy 2024-03-26T02:03:58.124 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:03:58 smithi019 ceph-mon[66359]: pgmap v232: 97 pgs: 97 active+clean; 336 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 4.7 MiB/s wr, 610 op/s; 306 KiB/s, 33 objects/s recovering 2024-03-26T02:03:58.127 INFO:teuthology.orchestra.run.smithi019.stdout: 5 267370 10.13 MB/sec execute 511 sec latency 55.560 ms 2024-03-26T02:03:58.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:03:58 smithi069 ceph-mon[53087]: pgmap v232: 97 pgs: 97 active+clean; 336 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 4.7 MiB/s wr, 610 op/s; 306 KiB/s, 33 objects/s recovering 2024-03-26T02:03:59.127 INFO:teuthology.orchestra.run.smithi019.stdout: 5 267697 10.13 MB/sec execute 512 sec latency 41.338 ms 2024-03-26T02:04:00.127 INFO:teuthology.orchestra.run.smithi019.stdout: 5 268066 10.13 MB/sec execute 513 sec latency 66.760 ms 2024-03-26T02:04:00.189 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:00 smithi019 ceph-mon[66359]: pgmap v233: 97 pgs: 97 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.5 MiB/s wr, 523 op/s; 56 KiB/s, 13 objects/s recovering 2024-03-26T02:04:00.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:00 smithi069 ceph-mon[53087]: pgmap v233: 97 pgs: 97 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.5 MiB/s wr, 523 op/s; 56 KiB/s, 13 objects/s recovering 2024-03-26T02:04:01.127 INFO:teuthology.orchestra.run.smithi019.stdout: 5 268356 10.12 MB/sec execute 514 sec latency 101.311 ms 2024-03-26T02:04:01.785 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:01 smithi019 ceph-mon[66359]: pgmap v234: 97 pgs: 97 active+clean; 335 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.9 MiB/s wr, 473 op/s; 43 KiB/s, 10 objects/s recovering 2024-03-26T02:04:01.786 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:01.786 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:04:01.786 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T02:04:02.019 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:01 smithi069 ceph-mon[53087]: pgmap v234: 97 pgs: 97 active+clean; 335 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.9 MiB/s wr, 473 op/s; 43 KiB/s, 10 objects/s recovering 2024-03-26T02:04:02.019 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:02.019 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:04:02.019 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T02:04:02.127 INFO:teuthology.orchestra.run.smithi019.stdout: 5 268697 10.13 MB/sec execute 515 sec latency 49.447 ms 2024-03-26T02:04:02.804 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:02 smithi019 ceph-mon[66359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T02:04:02.804 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:02 smithi019 ceph-mon[66359]: Upgrade: osd.5 is safe to restart 2024-03-26T02:04:03.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:02 smithi069 ceph-mon[53087]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T02:04:03.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:02 smithi069 ceph-mon[53087]: Upgrade: osd.5 is safe to restart 2024-03-26T02:04:03.127 INFO:teuthology.orchestra.run.smithi019.stdout: 5 268940 10.13 MB/sec execute 516 sec latency 84.331 ms 2024-03-26T02:04:04.076 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:03 smithi019 ceph-mon[66359]: pgmap v235: 97 pgs: 97 active+clean; 335 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.4 MiB/s wr, 443 op/s; 0 B/s, 7 objects/s recovering 2024-03-26T02:04:04.076 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:03 smithi019 ceph-mon[66359]: Upgrade: Updating osd.5 2024-03-26T02:04:04.076 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:03 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:04.076 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:03 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T02:04:04.076 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:03 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:04:04.076 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:03 smithi019 ceph-mon[66359]: Deploying daemon osd.5 on smithi069 2024-03-26T02:04:04.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:03 smithi069 ceph-mon[53087]: pgmap v235: 97 pgs: 97 active+clean; 335 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.4 MiB/s wr, 443 op/s; 0 B/s, 7 objects/s recovering 2024-03-26T02:04:04.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:03 smithi069 ceph-mon[53087]: Upgrade: Updating osd.5 2024-03-26T02:04:04.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:03 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:04.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:03 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T02:04:04.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:03 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:04:04.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:03 smithi069 ceph-mon[53087]: Deploying daemon osd.5 on smithi069 2024-03-26T02:04:04.127 INFO:teuthology.orchestra.run.smithi019.stdout: 5 269218 10.12 MB/sec execute 517 sec latency 61.723 ms 2024-03-26T02:04:05.043 INFO:teuthology.orchestra.run.smithi019.stdout:true 2024-03-26T02:04:05.127 INFO:teuthology.orchestra.run.smithi019.stdout: 5 269519 10.12 MB/sec execute 518 sec latency 45.330 ms 2024-03-26T02:04:05.949 INFO:teuthology.orchestra.run.smithi019.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:04:05.949 INFO:teuthology.orchestra.run.smithi019.stdout:alertmanager.smithi019 smithi019 *:9093,9094 running (8m) 63s ago 17m 14.5M - 0.23.0 ba2b418f427c a6630841c2b2 2024-03-26T02:04:05.949 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi019 smithi019 running (4m) 63s ago 17m 6660k - 19.0.0-1775-g5e70124b 5f21c9264136 c5f91a47ddf2 2024-03-26T02:04:05.949 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi069 smithi069 running (4m) 25s ago 16m 6669k - 19.0.0-1775-g5e70124b 5f21c9264136 05128803b49e 2024-03-26T02:04:05.949 INFO:teuthology.orchestra.run.smithi019.stdout:grafana.smithi019 smithi019 *:3000 running (7m) 63s ago 16m 46.9M - 8.3.5 dad864ee21e9 a97dea3b5fec 2024-03-26T02:04:05.949 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi019.lmryom smithi019 running (11m) 63s ago 11m 38.4M - 17.2.0 e1d6a67b021e 91a7c2942715 2024-03-26T02:04:05.949 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi069.spgwgo smithi069 running (11m) 25s ago 11m 15.1M - 17.2.0 e1d6a67b021e 94d96338dbe1 2024-03-26T02:04:05.949 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi019.ixuvdm smithi019 *:8443,9283,8765 running (6m) 63s ago 18m 508M - 19.0.0-1775-g5e70124b 5f21c9264136 4460b22c7a56 2024-03-26T02:04:05.949 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi069.wpbewu smithi069 *:8443,9283,8765 running (6m) 25s ago 16m 454M - 19.0.0-1775-g5e70124b 5f21c9264136 380fd33842a8 2024-03-26T02:04:05.949 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi019 smithi019 running (5m) 63s ago 18m 52.0M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 c18934705774 2024-03-26T02:04:05.949 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi069 smithi069 running (5m) 25s ago 16m 42.3M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 a856bd263104 2024-03-26T02:04:05.949 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.0.0.smithi019.qbizxo smithi019 *:2049 running (10m) 63s ago 10m 283M - 4.0 e1d6a67b021e cd6628a9075d 2024-03-26T02:04:05.949 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.1.0.smithi069.uawdhc smithi069 *:2049 running (10m) 25s ago 10m 85.7M - 4.0 e1d6a67b021e 6ee3695e7a10 2024-03-26T02:04:05.950 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi019 smithi019 *:9100 running (7m) 63s ago 16m 10.2M - 1.3.1 1dbe0e931976 0d1d184bec63 2024-03-26T02:04:05.950 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi069 smithi069 *:9100 running (7m) 25s ago 16m 19.2M - 1.3.1 1dbe0e931976 c49a006f94e7 2024-03-26T02:04:05.950 INFO:teuthology.orchestra.run.smithi019.stdout:osd.0 smithi019 running (3m) 63s ago 15m 340M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 3f931cfe320a 2024-03-26T02:04:05.950 INFO:teuthology.orchestra.run.smithi019.stdout:osd.1 smithi019 running (2m) 63s ago 15m 289M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 758caf4ed841 2024-03-26T02:04:05.950 INFO:teuthology.orchestra.run.smithi019.stdout:osd.2 smithi019 running (2m) 63s ago 14m 228M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 6a9b701517ca 2024-03-26T02:04:05.950 INFO:teuthology.orchestra.run.smithi019.stdout:osd.3 smithi019 running (73s) 63s ago 14m 12.6M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 0401940d882b 2024-03-26T02:04:05.950 INFO:teuthology.orchestra.run.smithi019.stdout:osd.4 smithi069 running (29s) 25s ago 13m 11.6M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 6eb3b2ce9978 2024-03-26T02:04:05.950 INFO:teuthology.orchestra.run.smithi019.stdout:osd.5 smithi069 running (13m) 25s ago 13m 623M 2704M 17.2.0 e1d6a67b021e 932e401d87c9 2024-03-26T02:04:05.950 INFO:teuthology.orchestra.run.smithi019.stdout:osd.6 smithi069 running (13m) 25s ago 13m 541M 2704M 17.2.0 e1d6a67b021e b123f75154c8 2024-03-26T02:04:05.950 INFO:teuthology.orchestra.run.smithi019.stdout:osd.7 smithi069 running (12m) 25s ago 12m 603M 2704M 17.2.0 e1d6a67b021e d175fbc9ff8a 2024-03-26T02:04:05.950 INFO:teuthology.orchestra.run.smithi019.stdout:prometheus.smithi019 smithi019 *:9095 running (6m) 63s ago 15m 67.6M - 2.33.4 514e6a882f6e 29f89db72a07 2024-03-26T02:04:05.984 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:05 smithi069 ceph-mon[53087]: pgmap v236: 97 pgs: 97 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.4 MiB/s wr, 414 op/s; 0 B/s, 6 objects/s recovering 2024-03-26T02:04:06.005 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:05 smithi019 ceph-mon[66359]: pgmap v236: 97 pgs: 97 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.4 MiB/s wr, 414 op/s; 0 B/s, 6 objects/s recovering 2024-03-26T02:04:06.006 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:05 smithi019 ceph-mon[66359]: from='client.34374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:04:06.006 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:06.127 INFO:teuthology.orchestra.run.smithi019.stdout: 5 269750 10.11 MB/sec execute 519 sec latency 85.739 ms 2024-03-26T02:04:06.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:05 smithi069 ceph-mon[53087]: from='client.34374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:04:06.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:06.533 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:04:06.534 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": { 2024-03-26T02:04:06.534 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:04:06.534 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:04:06.534 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": { 2024-03-26T02:04:06.534 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:04:06.534 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:04:06.534 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": { 2024-03-26T02:04:06.534 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-03-26T02:04:06.534 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 5 2024-03-26T02:04:06.534 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:04:06.534 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": { 2024-03-26T02:04:06.534 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:04:06.535 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:04:06.535 INFO:teuthology.orchestra.run.smithi019.stdout: "overall": { 2024-03-26T02:04:06.535 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-26T02:04:06.535 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 9 2024-03-26T02:04:06.535 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-26T02:04:06.535 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:04:07.028 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:06 smithi019 ceph-mon[66359]: from='client.34378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:04:07.029 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:06 smithi019 ceph-mon[66359]: from='client.34382 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:04:07.029 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:06 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/2944503153' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:04:07.083 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:04:07.083 INFO:teuthology.orchestra.run.smithi019.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2", 2024-03-26T02:04:07.083 INFO:teuthology.orchestra.run.smithi019.stdout: "in_progress": true, 2024-03-26T02:04:07.083 INFO:teuthology.orchestra.run.smithi019.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T02:04:07.083 INFO:teuthology.orchestra.run.smithi019.stdout: "services_complete": [ 2024-03-26T02:04:07.083 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr", 2024-03-26T02:04:07.084 INFO:teuthology.orchestra.run.smithi019.stdout: "crash", 2024-03-26T02:04:07.084 INFO:teuthology.orchestra.run.smithi019.stdout: "mon" 2024-03-26T02:04:07.084 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-26T02:04:07.084 INFO:teuthology.orchestra.run.smithi019.stdout: "progress": "11/23 daemons upgraded", 2024-03-26T02:04:07.084 INFO:teuthology.orchestra.run.smithi019.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T02:04:07.084 INFO:teuthology.orchestra.run.smithi019.stdout: "is_paused": false 2024-03-26T02:04:07.084 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:04:07.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:06 smithi069 ceph-mon[53087]: from='client.34378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:04:07.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:06 smithi069 ceph-mon[53087]: from='client.34382 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:04:07.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:06 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/2944503153' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:04:07.127 INFO:teuthology.orchestra.run.smithi019.stdout: 5 270026 10.10 MB/sec execute 520 sec latency 33.732 ms 2024-03-26T02:04:07.443 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:07 smithi069 systemd[1]: Stopping Ceph osd.5 for 692607c6-eb12-11ee-95d0-87774f69a715... 2024-03-26T02:04:07.725 INFO:teuthology.orchestra.run.smithi019.stdout:HEALTH_OK 2024-03-26T02:04:07.859 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:07 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[34284]: 2024-03-26T02:04:07.441+0000 7f134eb89700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:04:07.859 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:07 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[34284]: 2024-03-26T02:04:07.441+0000 7f134eb89700 -1 osd.5 139 *** Got signal Terminated *** 2024-03-26T02:04:07.859 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:07 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[34284]: 2024-03-26T02:04:07.441+0000 7f134eb89700 -1 osd.5 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:04:08.127 INFO:teuthology.orchestra.run.smithi019.stdout: 5 270261 10.09 MB/sec execute 521 sec latency 196.802 ms 2024-03-26T02:04:08.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:07 smithi019 ceph-mon[66359]: pgmap v237: 97 pgs: 97 active+clean; 340 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 4.6 MiB/s wr, 460 op/s; 0 B/s, 5 objects/s recovering 2024-03-26T02:04:08.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:07 smithi019 ceph-mon[66359]: from='client.34390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:04:08.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:07 smithi019 ceph-mon[66359]: osd.5 marked itself down and dead 2024-03-26T02:04:08.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:07 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/4285612981' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:04:08.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:07 smithi069 ceph-mon[53087]: pgmap v237: 97 pgs: 97 active+clean; 340 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 4.6 MiB/s wr, 460 op/s; 0 B/s, 5 objects/s recovering 2024-03-26T02:04:08.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:07 smithi069 ceph-mon[53087]: from='client.34390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:04:08.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:07 smithi069 ceph-mon[53087]: osd.5 marked itself down and dead 2024-03-26T02:04:08.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:07 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/4285612981' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:04:08.859 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:08 smithi069 podman[59831]: 2024-03-26 02:04:08.607774576 +0000 UTC m=+1.286026539 container died 932e401d87c95f4d278b47fb474b08f7156f4a4dd06c6ccd279319067390dadf (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, distribution-scope=public, ceph=True, name=centos-stream, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-03-26T02:04:09.127 INFO:teuthology.orchestra.run.smithi019.stdout: 5 270301 10.07 MB/sec execute 522 sec latency 1079.209 ms 2024-03-26T02:04:09.205 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:08 smithi019 ceph-mon[66359]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:04:09.206 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:08 smithi019 ceph-mon[66359]: failed to encode map e140 with expected crc 2024-03-26T02:04:09.206 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:08 smithi019 ceph-mon[66359]: failed to encode map e140 with expected crc 2024-03-26T02:04:09.206 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:08 smithi019 ceph-mon[66359]: failed to encode map e140 with expected crc 2024-03-26T02:04:09.206 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:08 smithi019 ceph-mon[66359]: osdmap e140: 8 total, 7 up, 8 in 2024-03-26T02:04:09.206 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:08 smithi019 ceph-mon[66359]: failed to encode map e140 with expected crc 2024-03-26T02:04:09.259 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:08 smithi069 ceph-mon[53087]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:04:09.260 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:08 smithi069 ceph-mon[53087]: failed to encode map e140 with expected crc 2024-03-26T02:04:09.260 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:08 smithi069 ceph-mon[53087]: failed to encode map e140 with expected crc 2024-03-26T02:04:09.260 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:08 smithi069 ceph-mon[53087]: failed to encode map e140 with expected crc 2024-03-26T02:04:09.260 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:08 smithi069 ceph-mon[53087]: osdmap e140: 8 total, 7 up, 8 in 2024-03-26T02:04:09.260 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:08 smithi069 ceph-mon[53087]: failed to encode map e140 with expected crc 2024-03-26T02:04:09.260 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:08 smithi069 podman[59831]: 2024-03-26 02:04:08.974658332 +0000 UTC m=+1.652910301 container cleanup 932e401d87c95f4d278b47fb474b08f7156f4a4dd06c6ccd279319067390dadf (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, vcs-type=git, com.redhat.component=centos-stream-container, ceph=True, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , version=8, GIT_BRANCH=HEAD, name=centos-stream) 2024-03-26T02:04:09.260 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:08 smithi069 bash[59831]: ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5 2024-03-26T02:04:09.609 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:09 smithi069 podman[59848]: 2024-03-26 02:04:09.259530381 +0000 UTC m=+0.652838271 container remove 932e401d87c95f4d278b47fb474b08f7156f4a4dd06c6ccd279319067390dadf (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, version=8, name=centos-stream, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, ceph=True, release=754, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux ) 2024-03-26T02:04:09.888 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:09 smithi069 podman[59940]: 2024-03-26 02:04:09.528442459 +0000 UTC m=+0.019648195 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:04:09.888 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:09 smithi069 podman[59940]: 2024-03-26 02:04:09.652031536 +0000 UTC m=+0.143237270 container create bec6c8094aefc1d2e3df95be4da0894b71f2726425ee3ca9bef404757212d2b0 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-26T02:04:09.889 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:09 smithi069 podman[59940]: 2024-03-26 02:04:09.888331127 +0000 UTC m=+0.379536859 container init bec6c8094aefc1d2e3df95be4da0894b71f2726425ee3ca9bef404757212d2b0 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-26T02:04:09.976 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:09 smithi019 ceph-mon[66359]: pgmap v239: 97 pgs: 5 peering, 10 stale+active+clean, 82 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.8 MiB/s wr, 465 op/s 2024-03-26T02:04:09.976 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:09 smithi019 ceph-mon[66359]: failed to encode map e141 with expected crc 2024-03-26T02:04:09.976 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:09 smithi019 ceph-mon[66359]: failed to encode map e141 with expected crc 2024-03-26T02:04:09.976 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:09 smithi019 ceph-mon[66359]: failed to encode map e141 with expected crc 2024-03-26T02:04:09.976 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:09 smithi019 ceph-mon[66359]: osdmap e141: 8 total, 7 up, 8 in 2024-03-26T02:04:09.976 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:09 smithi019 ceph-mon[66359]: failed to encode map e141 with expected crc 2024-03-26T02:04:09.976 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:09 smithi019 ceph-mon[66359]: failed to encode map e141 with expected crc 2024-03-26T02:04:09.976 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:09 smithi019 ceph-mon[66359]: failed to encode map e141 with expected crc 2024-03-26T02:04:10.127 INFO:teuthology.orchestra.run.smithi019.stdout: 5 270675 10.07 MB/sec execute 523 sec latency 61.444 ms 2024-03-26T02:04:10.205 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:09 smithi069 ceph-mon[53087]: pgmap v239: 97 pgs: 5 peering, 10 stale+active+clean, 82 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.8 MiB/s wr, 465 op/s 2024-03-26T02:04:10.205 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:09 smithi069 ceph-mon[53087]: failed to encode map e141 with expected crc 2024-03-26T02:04:10.205 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:09 smithi069 ceph-mon[53087]: failed to encode map e141 with expected crc 2024-03-26T02:04:10.205 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:09 smithi069 ceph-mon[53087]: failed to encode map e141 with expected crc 2024-03-26T02:04:10.205 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:09 smithi069 ceph-mon[53087]: osdmap e141: 8 total, 7 up, 8 in 2024-03-26T02:04:10.205 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:09 smithi069 ceph-mon[53087]: failed to encode map e141 with expected crc 2024-03-26T02:04:10.205 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:09 smithi069 ceph-mon[53087]: failed to encode map e141 with expected crc 2024-03-26T02:04:10.205 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:09 smithi069 ceph-mon[53087]: failed to encode map e141 with expected crc 2024-03-26T02:04:10.206 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:09 smithi069 podman[59940]: 2024-03-26 02:04:09.89223129 +0000 UTC m=+0.383437024 container start bec6c8094aefc1d2e3df95be4da0894b71f2726425ee3ca9bef404757212d2b0 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-26T02:04:10.206 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:10 smithi069 podman[59940]: 2024-03-26 02:04:10.013418543 +0000 UTC m=+0.504624285 container attach bec6c8094aefc1d2e3df95be4da0894b71f2726425ee3ca9bef404757212d2b0 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, GIT_BRANCH=HEAD) 2024-03-26T02:04:10.584 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:10 smithi069 podman[59940]: 2024-03-26 02:04:10.205638337 +0000 UTC m=+0.696844072 container died bec6c8094aefc1d2e3df95be4da0894b71f2726425ee3ca9bef404757212d2b0 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124) 2024-03-26T02:04:11.116 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:11 smithi069 ceph-mon[53087]: Health check failed: Degraded data redundancy: 18/1650 objects degraded (1.091%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T02:04:11.116 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:10 smithi069 podman[59966]: 2024-03-26 02:04:10.860006612 +0000 UTC m=+0.647488548 container remove bec6c8094aefc1d2e3df95be4da0894b71f2726425ee3ca9bef404757212d2b0 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:04:11.117 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:10 smithi069 systemd[1]: ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.5.service: Deactivated successfully. 2024-03-26T02:04:11.117 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:10 smithi069 systemd[1]: Stopped Ceph osd.5 for 692607c6-eb12-11ee-95d0-87774f69a715. 2024-03-26T02:04:11.117 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:10 smithi069 systemd[1]: ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.5.service: Consumed 5min 56.365s CPU time. 2024-03-26T02:04:11.127 INFO:teuthology.orchestra.run.smithi019.stdout: 5 271019 10.07 MB/sec execute 524 sec latency 33.263 ms 2024-03-26T02:04:11.235 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:11 smithi019 ceph-mon[66359]: Health check failed: Degraded data redundancy: 18/1650 objects degraded (1.091%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T02:04:11.377 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:11 smithi069 systemd[1]: Starting Ceph osd.5 for 692607c6-eb12-11ee-95d0-87774f69a715... 2024-03-26T02:04:11.378 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:11 smithi069 podman[60061]: 2024-03-26 02:04:11.259790442 +0000 UTC m=+0.024147686 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:04:11.689 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:11 smithi069 podman[60061]: 2024-03-26 02:04:11.377792761 +0000 UTC m=+0.142150000 container create 409becb849b2fe743d1cdcd0a82639873220eabce54c7356a6cd8cbb7216dab8 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5-activate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, GIT_CLEAN=True) 2024-03-26T02:04:11.690 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:11 smithi069 podman[60061]: 2024-03-26 02:04:11.689356058 +0000 UTC m=+0.453713308 container init 409becb849b2fe743d1cdcd0a82639873220eabce54c7356a6cd8cbb7216dab8 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5-activate, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319) 2024-03-26T02:04:12.006 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:11 smithi069 podman[60061]: 2024-03-26 02:04:11.694181469 +0000 UTC m=+0.458538707 container start 409becb849b2fe743d1cdcd0a82639873220eabce54c7356a6cd8cbb7216dab8 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5-activate, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:04:12.007 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:11 smithi069 podman[60061]: 2024-03-26 02:04:11.797607865 +0000 UTC m=+0.561965111 container attach 409becb849b2fe743d1cdcd0a82639873220eabce54c7356a6cd8cbb7216dab8 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5-activate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True) 2024-03-26T02:04:12.127 INFO:teuthology.orchestra.run.smithi019.stdout: 5 271351 10.06 MB/sec execute 525 sec latency 68.902 ms 2024-03-26T02:04:12.255 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:12 smithi019 ceph-mon[66359]: pgmap v241: 97 pgs: 2 active+undersized, 3 active+undersized+degraded, 5 peering, 9 stale+active+clean, 78 active+clean; 335 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 3.6 MiB/s wr, 343 op/s; 18/1650 objects degraded (1.091%) 2024-03-26T02:04:12.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:12 smithi069 ceph-mon[53087]: pgmap v241: 97 pgs: 2 active+undersized, 3 active+undersized+degraded, 5 peering, 9 stale+active+clean, 78 active+clean; 335 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 3.6 MiB/s wr, 343 op/s; 18/1650 objects degraded (1.091%) 2024-03-26T02:04:12.360 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:12 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5-activate[60076]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-26T02:04:12.360 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:12 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5-activate[60076]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-26T02:04:12.360 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:12 smithi069 bash[60061]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-26T02:04:12.360 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:12 smithi069 bash[60061]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-26T02:04:12.360 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:12 smithi069 bash[60061]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-26T02:04:12.360 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:12 smithi069 bash[60061]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-26T02:04:12.360 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:12 smithi069 bash[60061]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-26T02:04:12.360 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:12 smithi069 bash[60061]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-26T02:04:12.360 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:12 smithi069 bash[60061]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-26T02:04:12.360 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:12 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5-activate[60076]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-26T02:04:12.360 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:12 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5-activate[60076]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-26T02:04:12.360 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:12 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5-activate[60076]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-26T02:04:12.361 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:12 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5-activate[60076]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-26T02:04:12.361 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:12 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5-activate[60076]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-26T02:04:12.361 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:12 smithi069 podman[60061]: 2024-03-26 02:04:12.254312273 +0000 UTC m=+1.018669514 container died 409becb849b2fe743d1cdcd0a82639873220eabce54c7356a6cd8cbb7216dab8 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5-activate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-26T02:04:13.104 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:12 smithi069 podman[60192]: 2024-03-26 02:04:12.852905218 +0000 UTC m=+0.592435150 container remove 409becb849b2fe743d1cdcd0a82639873220eabce54c7356a6cd8cbb7216dab8 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5-activate, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-26T02:04:13.104 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:13 smithi069 podman[60254]: 2024-03-26 02:04:12.983807048 +0000 UTC m=+0.022331125 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:04:13.127 INFO:teuthology.orchestra.run.smithi019.stdout: 5 271734 10.07 MB/sec execute 526 sec latency 42.776 ms 2024-03-26T02:04:13.359 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:13 smithi069 podman[60254]: 2024-03-26 02:04:13.103978892 +0000 UTC m=+0.142502954 container create fb31f0e68fb7a960d4760fc5788d1aa49546d767311574fddb8b7e9bb5e437f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-26T02:04:13.359 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:13 smithi069 podman[60254]: 2024-03-26 02:04:13.315239186 +0000 UTC m=+0.353763251 container init fb31f0e68fb7a960d4760fc5788d1aa49546d767311574fddb8b7e9bb5e437f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=squid-5e70124) 2024-03-26T02:04:13.359 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:13 smithi069 podman[60254]: 2024-03-26 02:04:13.318555914 +0000 UTC m=+0.357079977 container start fb31f0e68fb7a960d4760fc5788d1aa49546d767311574fddb8b7e9bb5e437f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-26T02:04:13.360 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:13 smithi069 ceph-osd[60272]: -- 172.21.15.69:0/2394607269 <== mon.1 v2:172.21.15.69:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556af03a71e0 con 0x556af0378c00 2024-03-26T02:04:13.859 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:13 smithi069 bash[60254]: fb31f0e68fb7a960d4760fc5788d1aa49546d767311574fddb8b7e9bb5e437f2 2024-03-26T02:04:13.859 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:13 smithi069 systemd[1]: Started Ceph osd.5 for 692607c6-eb12-11ee-95d0-87774f69a715. 2024-03-26T02:04:14.042 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:14 smithi019 ceph-mon[66359]: pgmap v242: 97 pgs: 8 active+undersized, 18 active+undersized+degraded, 5 peering, 66 active+clean; 329 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 3.8 MiB/s wr, 406 op/s; 136/1584 objects degraded (8.586%); 834 B/s, 0 objects/s recovering 2024-03-26T02:04:14.045 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:14 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:14.045 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:14 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:14.046 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:14 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:04:14.127 INFO:teuthology.orchestra.run.smithi019.stdout: 5 272042 10.07 MB/sec execute 527 sec latency 85.540 ms 2024-03-26T02:04:14.359 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:13 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:04:13.863+0000 7fddb4fa27c0 -1 Falling back to public interface 2024-03-26T02:04:14.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:14 smithi069 ceph-mon[53087]: pgmap v242: 97 pgs: 8 active+undersized, 18 active+undersized+degraded, 5 peering, 66 active+clean; 329 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 3.8 MiB/s wr, 406 op/s; 136/1584 objects degraded (8.586%); 834 B/s, 0 objects/s recovering 2024-03-26T02:04:14.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:14 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:14.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:14 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:14.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:14 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:04:15.127 INFO:teuthology.orchestra.run.smithi019.stdout: 5 272367 10.07 MB/sec execute 528 sec latency 44.688 ms 2024-03-26T02:04:16.073 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:16 smithi019 ceph-mon[66359]: pgmap v243: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.1 KiB/s rd, 3.7 MiB/s wr, 372 op/s; 190/1620 objects degraded (11.728%); 834 B/s, 0 objects/s recovering 2024-03-26T02:04:16.073 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:16 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:16.073 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:16 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:04:16.127 INFO:teuthology.orchestra.run.smithi019.stdout: 5 272701 10.07 MB/sec execute 529 sec latency 43.573 ms 2024-03-26T02:04:16.321 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:16 smithi069 ceph-mon[53087]: pgmap v243: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.1 KiB/s rd, 3.7 MiB/s wr, 372 op/s; 190/1620 objects degraded (11.728%); 834 B/s, 0 objects/s recovering 2024-03-26T02:04:16.321 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:16 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:16.321 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:16 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:04:17.089 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:17 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 190/1668 objects degraded (11.391%), 23 pgs degraded (PG_DEGRADED) 2024-03-26T02:04:17.127 INFO:teuthology.orchestra.run.smithi019.stdout: 5 273060 10.07 MB/sec execute 530 sec latency 41.480 ms 2024-03-26T02:04:17.322 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:17 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 190/1668 objects degraded (11.391%), 23 pgs degraded (PG_DEGRADED) 2024-03-26T02:04:18.121 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:18 smithi019 ceph-mon[66359]: pgmap v244: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.8 MiB/s wr, 423 op/s; 190/1668 objects degraded (11.391%); 759 B/s, 0 objects/s recovering 2024-03-26T02:04:18.121 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:18 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:18.122 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:18 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:18.129 INFO:teuthology.orchestra.run.smithi019.stdout: 5 273367 10.06 MB/sec execute 531 sec latency 78.332 ms 2024-03-26T02:04:18.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:18 smithi069 ceph-mon[53087]: pgmap v244: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.8 MiB/s wr, 423 op/s; 190/1668 objects degraded (11.391%); 759 B/s, 0 objects/s recovering 2024-03-26T02:04:18.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:18 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:18.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:18 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:19.128 INFO:teuthology.orchestra.run.smithi019.stdout: 5 273753 10.06 MB/sec execute 532 sec latency 35.466 ms 2024-03-26T02:04:20.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:20 smithi069 ceph-mon[53087]: pgmap v245: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.6 MiB/s wr, 454 op/s; 190/1716 objects degraded (11.072%); 667 B/s, 0 objects/s recovering 2024-03-26T02:04:20.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:20 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:20.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:20 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:20.127 INFO:teuthology.orchestra.run.smithi019.stdout: 5 274246 10.06 MB/sec execute 533 sec latency 48.238 ms 2024-03-26T02:04:20.145 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:20 smithi019 ceph-mon[66359]: pgmap v245: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.6 MiB/s wr, 454 op/s; 190/1716 objects degraded (11.072%); 667 B/s, 0 objects/s recovering 2024-03-26T02:04:20.145 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:20 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:20.145 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:20 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:21.128 INFO:teuthology.orchestra.run.smithi019.stdout: 5 274609 10.06 MB/sec execute 534 sec latency 63.300 ms 2024-03-26T02:04:22.128 INFO:teuthology.orchestra.run.smithi019.stdout: 5 274923 10.06 MB/sec execute 535 sec latency 55.368 ms 2024-03-26T02:04:22.208 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:22 smithi019 ceph-mon[66359]: pgmap v246: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 337 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.3 KiB/s rd, 4.0 MiB/s wr, 392 op/s; 188/1692 objects degraded (11.111%); 566 B/s, 0 objects/s recovering 2024-03-26T02:04:22.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:22 smithi069 ceph-mon[53087]: pgmap v246: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 337 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.3 KiB/s rd, 4.0 MiB/s wr, 392 op/s; 188/1692 objects degraded (11.111%); 566 B/s, 0 objects/s recovering 2024-03-26T02:04:22.859 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:22 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:04:22.385+0000 7fddb4fa27c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-03-26T02:04:23.128 INFO:teuthology.orchestra.run.smithi019.stdout: 5 275255 10.06 MB/sec execute 536 sec latency 54.824 ms 2024-03-26T02:04:23.232 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:23 smithi019 ceph-mon[66359]: pgmap v247: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.2 MiB/s wr, 547 op/s; 187/1644 objects degraded (11.375%); 556 B/s, 0 objects/s recovering 2024-03-26T02:04:23.232 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:23 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 187/1644 objects degraded (11.375%), 23 pgs degraded (PG_DEGRADED) 2024-03-26T02:04:23.232 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:23.232 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:23.232 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:04:23.232 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:04:23.232 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:23.232 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:23.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:23 smithi069 ceph-mon[53087]: pgmap v247: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.2 MiB/s wr, 547 op/s; 187/1644 objects degraded (11.375%); 556 B/s, 0 objects/s recovering 2024-03-26T02:04:23.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:23 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 187/1644 objects degraded (11.375%), 23 pgs degraded (PG_DEGRADED) 2024-03-26T02:04:23.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:23.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:23.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:04:23.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:04:23.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:23.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:24.128 INFO:teuthology.orchestra.run.smithi019.stdout: 5 275597 10.06 MB/sec execute 537 sec latency 75.481 ms 2024-03-26T02:04:24.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:24 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:04:24.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:24 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:04:24.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:24 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:04:24.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:24 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:04:24.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:24 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-26T02:04:24.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:24 smithi019 ceph-mon[66359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-26T02:04:24.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:24 smithi019 ceph-mon[66359]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-26T02:04:24.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:24 smithi019 ceph-mon[66359]: from='osd.5 [v2:172.21.15.69:6808/2149399876,v1:172.21.15.69:6809/2149399876]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T02:04:24.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:24 smithi019 ceph-mon[66359]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T02:04:24.359 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:23 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:04:23.929+0000 7fddb4fa27c0 -1 osd.5 139 log_to_monitors true 2024-03-26T02:04:24.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:24 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:04:24.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:24 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:04:24.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:24 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:04:24.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:24 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:04:24.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:24 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-26T02:04:24.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:24 smithi069 ceph-mon[53087]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-26T02:04:24.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:24 smithi069 ceph-mon[53087]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-26T02:04:24.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:24 smithi069 ceph-mon[53087]: from='osd.5 [v2:172.21.15.69:6808/2149399876,v1:172.21.15.69:6809/2149399876]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T02:04:24.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:24 smithi069 ceph-mon[53087]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T02:04:25.109 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:04:24 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:04:24.608+0000 7fdda23f2700 -1 osd.5 139 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:04:25.128 INFO:teuthology.orchestra.run.smithi019.stdout: 5 275845 10.06 MB/sec execute 538 sec latency 133.940 ms 2024-03-26T02:04:25.296 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:25 smithi019 ceph-mon[66359]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T02:04:25.297 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:25 smithi019 ceph-mon[66359]: osdmap e142: 8 total, 7 up, 8 in 2024-03-26T02:04:25.297 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:25 smithi019 ceph-mon[66359]: failed to encode map e142 with expected crc 2024-03-26T02:04:25.297 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:25 smithi019 ceph-mon[66359]: from='osd.5 [v2:172.21.15.69:6808/2149399876,v1:172.21.15.69:6809/2149399876]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-03-26T02:04:25.297 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:25 smithi019 ceph-mon[66359]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-03-26T02:04:25.297 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:25 smithi019 ceph-mon[66359]: failed to encode map e142 with expected crc 2024-03-26T02:04:25.297 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:25 smithi019 ceph-mon[66359]: failed to encode map e142 with expected crc 2024-03-26T02:04:25.297 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:25 smithi019 ceph-mon[66359]: from='osd.5 ' entity='osd.5' 2024-03-26T02:04:25.297 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:25 smithi019 ceph-mon[66359]: pgmap v249: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 334 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.2 MiB/s wr, 522 op/s; 181/1605 objects degraded (11.277%) 2024-03-26T02:04:25.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:25 smithi069 ceph-mon[53087]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T02:04:25.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:25 smithi069 ceph-mon[53087]: osdmap e142: 8 total, 7 up, 8 in 2024-03-26T02:04:25.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:25 smithi069 ceph-mon[53087]: failed to encode map e142 with expected crc 2024-03-26T02:04:25.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:25 smithi069 ceph-mon[53087]: from='osd.5 [v2:172.21.15.69:6808/2149399876,v1:172.21.15.69:6809/2149399876]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-03-26T02:04:25.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:25 smithi069 ceph-mon[53087]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-03-26T02:04:25.611 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:25 smithi069 ceph-mon[53087]: failed to encode map e142 with expected crc 2024-03-26T02:04:25.611 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:25 smithi069 ceph-mon[53087]: failed to encode map e142 with expected crc 2024-03-26T02:04:25.611 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:25 smithi069 ceph-mon[53087]: from='osd.5 ' entity='osd.5' 2024-03-26T02:04:25.611 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:25 smithi069 ceph-mon[53087]: pgmap v249: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 334 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.2 MiB/s wr, 522 op/s; 181/1605 objects degraded (11.277%) 2024-03-26T02:04:26.128 INFO:teuthology.orchestra.run.smithi019.stdout: 5 276127 10.05 MB/sec execute 539 sec latency 331.747 ms 2024-03-26T02:04:26.313 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:26 smithi019 ceph-mon[66359]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:04:26.314 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:26 smithi019 ceph-mon[66359]: osd.5 [v2:172.21.15.69:6808/2149399876,v1:172.21.15.69:6809/2149399876] boot 2024-03-26T02:04:26.314 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:26 smithi019 ceph-mon[66359]: osdmap e143: 8 total, 8 up, 8 in 2024-03-26T02:04:26.314 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:26 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:04:26.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:26 smithi069 ceph-mon[53087]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:04:26.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:26 smithi069 ceph-mon[53087]: osd.5 [v2:172.21.15.69:6808/2149399876,v1:172.21.15.69:6809/2149399876] boot 2024-03-26T02:04:26.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:26 smithi069 ceph-mon[53087]: osdmap e143: 8 total, 8 up, 8 in 2024-03-26T02:04:26.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:26 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:04:27.128 INFO:teuthology.orchestra.run.smithi019.stdout: 5 276256 10.04 MB/sec execute 540 sec latency 1151.262 ms 2024-03-26T02:04:27.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:27 smithi019 ceph-mon[66359]: failed to encode map e143 with expected crc 2024-03-26T02:04:27.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:27 smithi019 ceph-mon[66359]: failed to encode map e143 with expected crc 2024-03-26T02:04:27.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:27 smithi019 ceph-mon[66359]: failed to encode map e143 with expected crc 2024-03-26T02:04:27.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:27 smithi019 ceph-mon[66359]: osdmap e144: 8 total, 8 up, 8 in 2024-03-26T02:04:27.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:27 smithi019 ceph-mon[66359]: pgmap v252: 97 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+degraded, 1 remapped+peering, 1 unknown, 7 active+undersized, 16 active+undersized+degraded, 70 active+clean; 337 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 5.2 MiB/s wr, 573 op/s; 151/1614 objects degraded (9.356%); 14 B/s, 0 objects/s recovering 2024-03-26T02:04:27.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:27 smithi069 ceph-mon[53087]: failed to encode map e143 with expected crc 2024-03-26T02:04:27.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:27 smithi069 ceph-mon[53087]: failed to encode map e143 with expected crc 2024-03-26T02:04:27.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:27 smithi069 ceph-mon[53087]: failed to encode map e143 with expected crc 2024-03-26T02:04:27.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:27 smithi069 ceph-mon[53087]: osdmap e144: 8 total, 8 up, 8 in 2024-03-26T02:04:27.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:27 smithi069 ceph-mon[53087]: pgmap v252: 97 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+degraded, 1 remapped+peering, 1 unknown, 7 active+undersized, 16 active+undersized+degraded, 70 active+clean; 337 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 5.2 MiB/s wr, 573 op/s; 151/1614 objects degraded (9.356%); 14 B/s, 0 objects/s recovering 2024-03-26T02:04:28.128 INFO:teuthology.orchestra.run.smithi019.stdout: 5 276385 10.03 MB/sec execute 541 sec latency 1661.958 ms 2024-03-26T02:04:28.156 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:28 smithi019 ceph-mon[66359]: failed to encode map e144 with expected crc 2024-03-26T02:04:28.156 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:28 smithi019 ceph-mon[66359]: failed to encode map e144 with expected crc 2024-03-26T02:04:28.156 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:28 smithi019 ceph-mon[66359]: failed to encode map e144 with expected crc 2024-03-26T02:04:28.156 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:28 smithi019 ceph-mon[66359]: failed to encode map e144 with expected crc 2024-03-26T02:04:28.156 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:28 smithi019 ceph-mon[66359]: failed to encode map e144 with expected crc 2024-03-26T02:04:28.156 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:28 smithi019 ceph-mon[66359]: failed to encode map e144 with expected crc 2024-03-26T02:04:28.156 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:28 smithi019 ceph-mon[66359]: failed to encode map e144 with expected crc 2024-03-26T02:04:28.156 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:28 smithi019 ceph-mon[66359]: failed to encode map e144 with expected crc 2024-03-26T02:04:28.156 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:28 smithi019 ceph-mon[66359]: failed to encode map e144 with expected crc 2024-03-26T02:04:28.157 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:28 smithi019 ceph-mon[66359]: osdmap e145: 8 total, 8 up, 8 in 2024-03-26T02:04:28.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:28 smithi069 ceph-mon[53087]: failed to encode map e144 with expected crc 2024-03-26T02:04:28.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:28 smithi069 ceph-mon[53087]: failed to encode map e144 with expected crc 2024-03-26T02:04:28.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:28 smithi069 ceph-mon[53087]: failed to encode map e144 with expected crc 2024-03-26T02:04:28.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:28 smithi069 ceph-mon[53087]: failed to encode map e144 with expected crc 2024-03-26T02:04:28.611 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:28 smithi069 ceph-mon[53087]: failed to encode map e144 with expected crc 2024-03-26T02:04:28.611 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:28 smithi069 ceph-mon[53087]: failed to encode map e144 with expected crc 2024-03-26T02:04:28.611 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:28 smithi069 ceph-mon[53087]: failed to encode map e144 with expected crc 2024-03-26T02:04:28.611 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:28 smithi069 ceph-mon[53087]: failed to encode map e144 with expected crc 2024-03-26T02:04:28.611 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:28 smithi069 ceph-mon[53087]: failed to encode map e144 with expected crc 2024-03-26T02:04:28.611 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:28 smithi069 ceph-mon[53087]: osdmap e145: 8 total, 8 up, 8 in 2024-03-26T02:04:29.128 INFO:teuthology.orchestra.run.smithi019.stdout: 5 276667 10.03 MB/sec execute 542 sec latency 380.334 ms 2024-03-26T02:04:29.170 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:29 smithi019 ceph-mon[66359]: failed to encode map e145 with expected crc 2024-03-26T02:04:29.170 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:29 smithi019 ceph-mon[66359]: failed to encode map e145 with expected crc 2024-03-26T02:04:29.170 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:29 smithi019 ceph-mon[66359]: failed to encode map e145 with expected crc 2024-03-26T02:04:29.170 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:29 smithi019 ceph-mon[66359]: osdmap e146: 8 total, 8 up, 8 in 2024-03-26T02:04:29.170 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:29 smithi019 ceph-mon[66359]: pgmap v255: 97 pgs: 2 active+recovery_wait+degraded, 1 active+recovering+degraded, 3 remapped+peering, 1 unknown, 5 active+undersized, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+undersized+degraded, 73 active+clean; 341 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 5.0 MiB/s wr, 417 op/s; 123/1659 objects degraded (7.414%); 46 KiB/s, 7 objects/s recovering 2024-03-26T02:04:29.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:29 smithi069 ceph-mon[53087]: failed to encode map e145 with expected crc 2024-03-26T02:04:29.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:29 smithi069 ceph-mon[53087]: failed to encode map e145 with expected crc 2024-03-26T02:04:29.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:29 smithi069 ceph-mon[53087]: failed to encode map e145 with expected crc 2024-03-26T02:04:29.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:29 smithi069 ceph-mon[53087]: osdmap e146: 8 total, 8 up, 8 in 2024-03-26T02:04:29.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:29 smithi069 ceph-mon[53087]: pgmap v255: 97 pgs: 2 active+recovery_wait+degraded, 1 active+recovering+degraded, 3 remapped+peering, 1 unknown, 5 active+undersized, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+undersized+degraded, 73 active+clean; 341 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 5.0 MiB/s wr, 417 op/s; 123/1659 objects degraded (7.414%); 46 KiB/s, 7 objects/s recovering 2024-03-26T02:04:30.128 INFO:teuthology.orchestra.run.smithi019.stdout: 5 276847 10.01 MB/sec execute 543 sec latency 908.492 ms 2024-03-26T02:04:30.182 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:30 smithi019 ceph-mon[66359]: failed to encode map e146 with expected crc 2024-03-26T02:04:30.182 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:30 smithi019 ceph-mon[66359]: failed to encode map e146 with expected crc 2024-03-26T02:04:30.182 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:30 smithi019 ceph-mon[66359]: failed to encode map e146 with expected crc 2024-03-26T02:04:30.182 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:30 smithi019 ceph-mon[66359]: failed to encode map e146 with expected crc 2024-03-26T02:04:30.182 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:30 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 151/1614 objects degraded (9.356%), 18 pgs degraded (PG_DEGRADED) 2024-03-26T02:04:30.182 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:30 smithi019 ceph-mon[66359]: osdmap e147: 8 total, 8 up, 8 in 2024-03-26T02:04:30.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:30 smithi069 ceph-mon[53087]: failed to encode map e146 with expected crc 2024-03-26T02:04:30.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:30 smithi069 ceph-mon[53087]: failed to encode map e146 with expected crc 2024-03-26T02:04:30.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:30 smithi069 ceph-mon[53087]: failed to encode map e146 with expected crc 2024-03-26T02:04:30.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:30 smithi069 ceph-mon[53087]: failed to encode map e146 with expected crc 2024-03-26T02:04:30.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:30 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 151/1614 objects degraded (9.356%), 18 pgs degraded (PG_DEGRADED) 2024-03-26T02:04:30.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:30 smithi069 ceph-mon[53087]: osdmap e147: 8 total, 8 up, 8 in 2024-03-26T02:04:31.128 INFO:teuthology.orchestra.run.smithi019.stdout: 5 277180 10.01 MB/sec execute 544 sec latency 35.671 ms 2024-03-26T02:04:31.200 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:31 smithi019 ceph-mon[66359]: failed to encode map e147 with expected crc 2024-03-26T02:04:31.200 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:31 smithi019 ceph-mon[66359]: failed to encode map e147 with expected crc 2024-03-26T02:04:31.200 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:31 smithi019 ceph-mon[66359]: failed to encode map e147 with expected crc 2024-03-26T02:04:31.200 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:31 smithi019 ceph-mon[66359]: failed to encode map e147 with expected crc 2024-03-26T02:04:31.200 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:31 smithi019 ceph-mon[66359]: failed to encode map e147 with expected crc 2024-03-26T02:04:31.200 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:31 smithi019 ceph-mon[66359]: pgmap v257: 97 pgs: 3 peering, 2 active+recovery_wait+degraded, 1 active+recovering+degraded, 2 remapped+peering, 1 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 337 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.6 KiB/s rd, 5.5 MiB/s wr, 433 op/s; 61/1656 objects degraded (3.684%); 492 KiB/s, 32 objects/s recovering 2024-03-26T02:04:31.200 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:31 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:31.200 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:31 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:04:31.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:31 smithi069 ceph-mon[53087]: failed to encode map e147 with expected crc 2024-03-26T02:04:31.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:31 smithi069 ceph-mon[53087]: failed to encode map e147 with expected crc 2024-03-26T02:04:31.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:31 smithi069 ceph-mon[53087]: failed to encode map e147 with expected crc 2024-03-26T02:04:31.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:31 smithi069 ceph-mon[53087]: failed to encode map e147 with expected crc 2024-03-26T02:04:31.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:31 smithi069 ceph-mon[53087]: failed to encode map e147 with expected crc 2024-03-26T02:04:31.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:31 smithi069 ceph-mon[53087]: pgmap v257: 97 pgs: 3 peering, 2 active+recovery_wait+degraded, 1 active+recovering+degraded, 2 remapped+peering, 1 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 337 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.6 KiB/s rd, 5.5 MiB/s wr, 433 op/s; 61/1656 objects degraded (3.684%); 492 KiB/s, 32 objects/s recovering 2024-03-26T02:04:31.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:31 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:31.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:31 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:04:32.128 INFO:teuthology.orchestra.run.smithi019.stdout: 5 277467 10.00 MB/sec execute 545 sec latency 67.886 ms 2024-03-26T02:04:33.128 INFO:teuthology.orchestra.run.smithi019.stdout: 5 277863 10.00 MB/sec execute 546 sec latency 42.964 ms 2024-03-26T02:04:33.998 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:33 smithi019 ceph-mon[66359]: pgmap v258: 97 pgs: 3 peering, 1 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 329 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 4.2 MiB/s wr, 335 op/s; 23/1623 objects degraded (1.417%); 1.5 MiB/s, 34 objects/s recovering 2024-03-26T02:04:34.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:33 smithi069 ceph-mon[53087]: pgmap v258: 97 pgs: 3 peering, 1 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 329 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 4.2 MiB/s wr, 335 op/s; 23/1623 objects degraded (1.417%); 1.5 MiB/s, 34 objects/s recovering 2024-03-26T02:04:34.128 INFO:teuthology.orchestra.run.smithi019.stdout: 5 278236 9.99 MB/sec execute 547 sec latency 40.642 ms 2024-03-26T02:04:35.027 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:34 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 23/1623 objects degraded (1.417%), 1 pg degraded (PG_DEGRADED) 2024-03-26T02:04:35.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:34 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 23/1623 objects degraded (1.417%), 1 pg degraded (PG_DEGRADED) 2024-03-26T02:04:35.128 INFO:teuthology.orchestra.run.smithi019.stdout: 5 278576 10.00 MB/sec execute 548 sec latency 50.888 ms 2024-03-26T02:04:36.053 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:35 smithi019 ceph-mon[66359]: pgmap v259: 97 pgs: 97 active+clean; 333 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 4.1 MiB/s wr, 502 op/s; 1.5 MiB/s, 35 objects/s recovering 2024-03-26T02:04:36.054 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:35 smithi019 ceph-mon[66359]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 23/1623 objects degraded (1.417%), 1 pg degraded) 2024-03-26T02:04:36.054 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:35 smithi019 ceph-mon[66359]: Cluster is now healthy 2024-03-26T02:04:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:35 smithi069 ceph-mon[53087]: pgmap v259: 97 pgs: 97 active+clean; 333 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 4.1 MiB/s wr, 502 op/s; 1.5 MiB/s, 35 objects/s recovering 2024-03-26T02:04:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:35 smithi069 ceph-mon[53087]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 23/1623 objects degraded (1.417%), 1 pg degraded) 2024-03-26T02:04:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:35 smithi069 ceph-mon[53087]: Cluster is now healthy 2024-03-26T02:04:36.128 INFO:teuthology.orchestra.run.smithi019.stdout: 5 279004 10.01 MB/sec execute 549 sec latency 44.156 ms 2024-03-26T02:04:37.128 INFO:teuthology.orchestra.run.smithi019.stdout: 5 279339 10.01 MB/sec execute 550 sec latency 37.906 ms 2024-03-26T02:04:37.854 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:37 smithi019 ceph-mon[66359]: pgmap v260: 97 pgs: 97 active+clean; 331 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 3.4 MiB/s wr, 350 op/s; 1.3 MiB/s, 28 objects/s recovering 2024-03-26T02:04:38.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:37 smithi069 ceph-mon[53087]: pgmap v260: 97 pgs: 97 active+clean; 331 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 3.4 MiB/s wr, 350 op/s; 1.3 MiB/s, 28 objects/s recovering 2024-03-26T02:04:38.128 INFO:teuthology.orchestra.run.smithi019.stdout: 5 279641 10.01 MB/sec execute 551 sec latency 45.613 ms 2024-03-26T02:04:38.205 INFO:teuthology.orchestra.run.smithi019.stdout:true 2024-03-26T02:04:38.882 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:38 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-26T02:04:39.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:38 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-26T02:04:39.113 INFO:teuthology.orchestra.run.smithi019.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:04:39.113 INFO:teuthology.orchestra.run.smithi019.stdout:alertmanager.smithi019 smithi019 *:9093,9094 running (8m) 96s ago 17m 14.5M - 0.23.0 ba2b418f427c a6630841c2b2 2024-03-26T02:04:39.113 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi019 smithi019 running (4m) 96s ago 17m 6660k - 19.0.0-1775-g5e70124b 5f21c9264136 c5f91a47ddf2 2024-03-26T02:04:39.113 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi069 smithi069 running (4m) 21s ago 17m 6669k - 19.0.0-1775-g5e70124b 5f21c9264136 05128803b49e 2024-03-26T02:04:39.113 INFO:teuthology.orchestra.run.smithi019.stdout:grafana.smithi019 smithi019 *:3000 running (8m) 96s ago 17m 46.9M - 8.3.5 dad864ee21e9 a97dea3b5fec 2024-03-26T02:04:39.113 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi019.lmryom smithi019 running (11m) 96s ago 11m 38.4M - 17.2.0 e1d6a67b021e 91a7c2942715 2024-03-26T02:04:39.113 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi069.spgwgo smithi069 running (11m) 21s ago 11m 15.2M - 17.2.0 e1d6a67b021e 94d96338dbe1 2024-03-26T02:04:39.113 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi019.ixuvdm smithi019 *:8443,9283,8765 running (7m) 96s ago 19m 508M - 19.0.0-1775-g5e70124b 5f21c9264136 4460b22c7a56 2024-03-26T02:04:39.114 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi069.wpbewu smithi069 *:8443,9283,8765 running (6m) 21s ago 17m 454M - 19.0.0-1775-g5e70124b 5f21c9264136 380fd33842a8 2024-03-26T02:04:39.114 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi019 smithi019 running (6m) 96s ago 19m 52.0M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 c18934705774 2024-03-26T02:04:39.114 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi069 smithi069 running (5m) 21s ago 17m 42.2M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 a856bd263104 2024-03-26T02:04:39.114 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.0.0.smithi019.qbizxo smithi019 *:2049 running (11m) 96s ago 11m 283M - 4.0 e1d6a67b021e cd6628a9075d 2024-03-26T02:04:39.114 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.1.0.smithi069.uawdhc smithi069 *:2049 running (11m) 21s ago 11m 85.8M - 4.0 e1d6a67b021e 6ee3695e7a10 2024-03-26T02:04:39.114 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi019 smithi019 *:9100 running (8m) 96s ago 17m 10.2M - 1.3.1 1dbe0e931976 0d1d184bec63 2024-03-26T02:04:39.114 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi069 smithi069 *:9100 running (7m) 21s ago 16m 19.2M - 1.3.1 1dbe0e931976 c49a006f94e7 2024-03-26T02:04:39.114 INFO:teuthology.orchestra.run.smithi019.stdout:osd.0 smithi019 running (4m) 96s ago 16m 340M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 3f931cfe320a 2024-03-26T02:04:39.114 INFO:teuthology.orchestra.run.smithi019.stdout:osd.1 smithi019 running (3m) 96s ago 15m 289M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 758caf4ed841 2024-03-26T02:04:39.114 INFO:teuthology.orchestra.run.smithi019.stdout:osd.2 smithi019 running (2m) 96s ago 15m 228M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 6a9b701517ca 2024-03-26T02:04:39.114 INFO:teuthology.orchestra.run.smithi019.stdout:osd.3 smithi019 running (106s) 96s ago 14m 12.6M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 0401940d882b 2024-03-26T02:04:39.114 INFO:teuthology.orchestra.run.smithi019.stdout:osd.4 smithi069 running (63s) 21s ago 14m 255M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 6eb3b2ce9978 2024-03-26T02:04:39.114 INFO:teuthology.orchestra.run.smithi019.stdout:osd.5 smithi069 running (26s) 21s ago 14m 12.6M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 fb31f0e68fb7 2024-03-26T02:04:39.115 INFO:teuthology.orchestra.run.smithi019.stdout:osd.6 smithi069 running (13m) 21s ago 13m 576M 2704M 17.2.0 e1d6a67b021e b123f75154c8 2024-03-26T02:04:39.115 INFO:teuthology.orchestra.run.smithi019.stdout:osd.7 smithi069 running (13m) 21s ago 13m 649M 2704M 17.2.0 e1d6a67b021e d175fbc9ff8a 2024-03-26T02:04:39.115 INFO:teuthology.orchestra.run.smithi019.stdout:prometheus.smithi019 smithi019 *:9095 running (6m) 96s ago 16m 67.6M - 2.33.4 514e6a882f6e 29f89db72a07 2024-03-26T02:04:39.128 INFO:teuthology.orchestra.run.smithi019.stdout: 5 279897 10.00 MB/sec execute 552 sec latency 70.217 ms 2024-03-26T02:04:39.760 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:04:39.760 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": { 2024-03-26T02:04:39.760 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:04:39.760 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:04:39.760 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": { 2024-03-26T02:04:39.760 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:04:39.760 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:04:39.760 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": { 2024-03-26T02:04:39.761 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-03-26T02:04:39.761 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 6 2024-03-26T02:04:39.761 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:04:39.761 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": { 2024-03-26T02:04:39.761 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:04:39.761 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:04:39.761 INFO:teuthology.orchestra.run.smithi019.stdout: "overall": { 2024-03-26T02:04:39.761 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-03-26T02:04:39.761 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 10 2024-03-26T02:04:39.761 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-26T02:04:39.761 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:04:39.890 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:39 smithi019 ceph-mon[66359]: from='client.34400 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:04:39.890 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:39 smithi019 ceph-mon[66359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-26T02:04:39.890 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:39 smithi019 ceph-mon[66359]: Upgrade: osd.6 is safe to restart 2024-03-26T02:04:39.890 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:39 smithi019 ceph-mon[66359]: from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:04:39.890 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:39 smithi019 ceph-mon[66359]: pgmap v261: 97 pgs: 97 active+clean; 336 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.2 MiB/s wr, 427 op/s; 1.0 MiB/s, 22 objects/s recovering 2024-03-26T02:04:39.890 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:39 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/3730746899' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:04:40.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:39 smithi069 ceph-mon[53087]: from='client.34400 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:04:40.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:39 smithi069 ceph-mon[53087]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-26T02:04:40.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:39 smithi069 ceph-mon[53087]: Upgrade: osd.6 is safe to restart 2024-03-26T02:04:40.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:39 smithi069 ceph-mon[53087]: from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:04:40.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:39 smithi069 ceph-mon[53087]: pgmap v261: 97 pgs: 97 active+clean; 336 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.2 MiB/s wr, 427 op/s; 1.0 MiB/s, 22 objects/s recovering 2024-03-26T02:04:40.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:39 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/3730746899' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:04:40.128 INFO:teuthology.orchestra.run.smithi019.stdout: 5 280219 10.00 MB/sec execute 553 sec latency 55.534 ms 2024-03-26T02:04:40.315 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:04:40.315 INFO:teuthology.orchestra.run.smithi019.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2", 2024-03-26T02:04:40.315 INFO:teuthology.orchestra.run.smithi019.stdout: "in_progress": true, 2024-03-26T02:04:40.315 INFO:teuthology.orchestra.run.smithi019.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T02:04:40.315 INFO:teuthology.orchestra.run.smithi019.stdout: "services_complete": [ 2024-03-26T02:04:40.315 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr", 2024-03-26T02:04:40.315 INFO:teuthology.orchestra.run.smithi019.stdout: "crash", 2024-03-26T02:04:40.316 INFO:teuthology.orchestra.run.smithi019.stdout: "mon" 2024-03-26T02:04:40.316 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-26T02:04:40.316 INFO:teuthology.orchestra.run.smithi019.stdout: "progress": "12/23 daemons upgraded", 2024-03-26T02:04:40.316 INFO:teuthology.orchestra.run.smithi019.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T02:04:40.316 INFO:teuthology.orchestra.run.smithi019.stdout: "is_paused": false 2024-03-26T02:04:40.316 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:04:40.907 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:40 smithi019 ceph-mon[66359]: from='client.44325 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:04:40.907 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:40.907 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T02:04:40.907 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:04:40.977 INFO:teuthology.orchestra.run.smithi019.stdout:HEALTH_OK 2024-03-26T02:04:41.074 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:40 smithi069 ceph-mon[53087]: from='client.44325 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:04:41.074 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:41.074 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T02:04:41.074 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:04:41.128 INFO:teuthology.orchestra.run.smithi019.stdout: 5 280524 9.99 MB/sec execute 554 sec latency 44.583 ms 2024-03-26T02:04:41.928 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:41 smithi019 ceph-mon[66359]: Upgrade: Updating osd.6 2024-03-26T02:04:41.928 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:41 smithi019 ceph-mon[66359]: Deploying daemon osd.6 on smithi069 2024-03-26T02:04:41.928 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:41 smithi019 ceph-mon[66359]: from='client.44333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:04:41.928 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:41 smithi019 ceph-mon[66359]: pgmap v262: 97 pgs: 97 active+clean; 334 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.0 MiB/s wr, 398 op/s; 908 KiB/s, 13 objects/s recovering 2024-03-26T02:04:41.928 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:41 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:41.928 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:41 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/496217923' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:04:42.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:41 smithi069 ceph-mon[53087]: Upgrade: Updating osd.6 2024-03-26T02:04:42.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:41 smithi069 ceph-mon[53087]: Deploying daemon osd.6 on smithi069 2024-03-26T02:04:42.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:41 smithi069 ceph-mon[53087]: from='client.44333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:04:42.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:41 smithi069 ceph-mon[53087]: pgmap v262: 97 pgs: 97 active+clean; 334 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.0 MiB/s wr, 398 op/s; 908 KiB/s, 13 objects/s recovering 2024-03-26T02:04:42.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:41 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:42.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:41 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/496217923' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:04:42.128 INFO:teuthology.orchestra.run.smithi019.stdout: 5 280870 9.98 MB/sec execute 555 sec latency 47.413 ms 2024-03-26T02:04:43.128 INFO:teuthology.orchestra.run.smithi019.stdout: 5 281231 9.98 MB/sec execute 556 sec latency 48.294 ms 2024-03-26T02:04:43.972 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:43 smithi019 ceph-mon[66359]: pgmap v263: 97 pgs: 97 active+clean; 339 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.0 MiB/s wr, 536 op/s; 733 KiB/s, 10 objects/s recovering 2024-03-26T02:04:44.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:43 smithi069 ceph-mon[53087]: pgmap v263: 97 pgs: 97 active+clean; 339 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.0 MiB/s wr, 536 op/s; 733 KiB/s, 10 objects/s recovering 2024-03-26T02:04:44.110 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:43 smithi069 systemd[1]: Stopping Ceph osd.6 for 692607c6-eb12-11ee-95d0-87774f69a715... 2024-03-26T02:04:44.128 INFO:teuthology.orchestra.run.smithi019.stdout: 5 281704 9.98 MB/sec execute 557 sec latency 47.616 ms 2024-03-26T02:04:44.609 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:44 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[37326]: 2024-03-26T02:04:44.227+0000 7f057499d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:04:44.609 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:44 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[37326]: 2024-03-26T02:04:44.227+0000 7f057499d700 -1 osd.6 147 *** Got signal Terminated *** 2024-03-26T02:04:44.609 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:44 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[37326]: 2024-03-26T02:04:44.227+0000 7f057499d700 -1 osd.6 147 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:04:45.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:44 smithi019 ceph-mon[66359]: osd.6 marked itself down and dead 2024-03-26T02:04:45.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:44 smithi069 ceph-mon[53087]: osd.6 marked itself down and dead 2024-03-26T02:04:45.128 INFO:teuthology.orchestra.run.smithi019.stdout: 5 282073 9.99 MB/sec execute 558 sec latency 262.342 ms 2024-03-26T02:04:45.828 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:45 smithi069 podman[62045]: 2024-03-26 02:04:45.510512441 +0000 UTC m=+1.396818482 container died b123f75154c81ce34a01e6623fdab44876fb1e16292ec3713a0d38a1e4a89f6c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, io.openshift.expose-services=, com.redhat.component=centos-stream-container, name=centos-stream, RELEASE=HEAD, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., version=8, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True) 2024-03-26T02:04:46.106 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:45 smithi019 ceph-mon[66359]: pgmap v264: 97 pgs: 97 active+clean; 328 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.9 MiB/s wr, 534 op/s; 144 KiB/s, 3 objects/s recovering 2024-03-26T02:04:46.106 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:45 smithi019 ceph-mon[66359]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:04:46.106 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:45 smithi019 ceph-mon[66359]: failed to encode map e148 with expected crc 2024-03-26T02:04:46.106 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:45 smithi019 ceph-mon[66359]: osdmap e148: 8 total, 7 up, 8 in 2024-03-26T02:04:46.106 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:45 smithi019 ceph-mon[66359]: failed to encode map e148 with expected crc 2024-03-26T02:04:46.106 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:45 smithi019 ceph-mon[66359]: failed to encode map e148 with expected crc 2024-03-26T02:04:46.106 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:46.106 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:04:46.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:45 smithi069 ceph-mon[53087]: pgmap v264: 97 pgs: 97 active+clean; 328 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.9 MiB/s wr, 534 op/s; 144 KiB/s, 3 objects/s recovering 2024-03-26T02:04:46.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:45 smithi069 ceph-mon[53087]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:04:46.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:45 smithi069 ceph-mon[53087]: failed to encode map e148 with expected crc 2024-03-26T02:04:46.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:45 smithi069 ceph-mon[53087]: osdmap e148: 8 total, 7 up, 8 in 2024-03-26T02:04:46.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:45 smithi069 ceph-mon[53087]: failed to encode map e148 with expected crc 2024-03-26T02:04:46.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:45 smithi069 ceph-mon[53087]: failed to encode map e148 with expected crc 2024-03-26T02:04:46.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:46.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:04:46.110 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:45 smithi069 podman[62045]: 2024-03-26 02:04:45.927612809 +0000 UTC m=+1.813918866 container cleanup b123f75154c81ce34a01e6623fdab44876fb1e16292ec3713a0d38a1e4a89f6c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6, release=754, RELEASE=HEAD, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, vcs-type=git, name=centos-stream, ceph=True) 2024-03-26T02:04:46.110 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:45 smithi069 bash[62045]: ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6 2024-03-26T02:04:46.128 INFO:teuthology.orchestra.run.smithi019.stdout: 5 282205 9.98 MB/sec execute 559 sec latency 1041.424 ms 2024-03-26T02:04:46.609 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:46 smithi069 podman[62062]: 2024-03-26 02:04:46.162756629 +0000 UTC m=+0.649495374 container remove b123f75154c81ce34a01e6623fdab44876fb1e16292ec3713a0d38a1e4a89f6c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, io.openshift.tags=base centos centos-stream, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, RELEASE=HEAD, vcs-type=git, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, ceph=True, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8) 2024-03-26T02:04:46.868 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:46 smithi019 ceph-mon[66359]: failed to encode map e149 with expected crc 2024-03-26T02:04:46.868 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:46 smithi019 ceph-mon[66359]: failed to encode map e149 with expected crc 2024-03-26T02:04:46.868 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:46 smithi019 ceph-mon[66359]: osdmap e149: 8 total, 7 up, 8 in 2024-03-26T02:04:46.868 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:46 smithi019 ceph-mon[66359]: failed to encode map e149 with expected crc 2024-03-26T02:04:46.868 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:46 smithi019 ceph-mon[66359]: failed to encode map e149 with expected crc 2024-03-26T02:04:47.000 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:46 smithi069 ceph-mon[53087]: failed to encode map e149 with expected crc 2024-03-26T02:04:47.000 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:46 smithi069 ceph-mon[53087]: failed to encode map e149 with expected crc 2024-03-26T02:04:47.000 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:46 smithi069 ceph-mon[53087]: osdmap e149: 8 total, 7 up, 8 in 2024-03-26T02:04:47.000 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:46 smithi069 ceph-mon[53087]: failed to encode map e149 with expected crc 2024-03-26T02:04:47.000 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:46 smithi069 ceph-mon[53087]: failed to encode map e149 with expected crc 2024-03-26T02:04:47.000 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:46 smithi069 podman[62184]: 2024-03-26 02:04:46.562287746 +0000 UTC m=+0.021499780 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:04:47.000 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:46 smithi069 podman[62184]: 2024-03-26 02:04:46.688777793 +0000 UTC m=+0.147989851 container create 444bbb4f535c346b59a45a52f5323f3e734a200b86d2f57c4373a75f4af592d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:04:47.001 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:46 smithi069 podman[62184]: 2024-03-26 02:04:46.899923724 +0000 UTC m=+0.359135734 container init 444bbb4f535c346b59a45a52f5323f3e734a200b86d2f57c4373a75f4af592d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:04:47.001 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:46 smithi069 podman[62184]: 2024-03-26 02:04:46.904493177 +0000 UTC m=+0.363705183 container start 444bbb4f535c346b59a45a52f5323f3e734a200b86d2f57c4373a75f4af592d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-26T02:04:47.001 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:47 smithi069 podman[62184]: 2024-03-26 02:04:47.000047184 +0000 UTC m=+0.459259204 container attach 444bbb4f535c346b59a45a52f5323f3e734a200b86d2f57c4373a75f4af592d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True) 2024-03-26T02:04:47.128 INFO:teuthology.orchestra.run.smithi019.stdout: 5 282586 9.99 MB/sec execute 560 sec latency 50.070 ms 2024-03-26T02:04:47.359 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:47 smithi069 podman[62184]: 2024-03-26 02:04:47.225590099 +0000 UTC m=+0.684802104 container died 444bbb4f535c346b59a45a52f5323f3e734a200b86d2f57c4373a75f4af592d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6-deactivate, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-26T02:04:47.883 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:47 smithi019 ceph-mon[66359]: pgmap v267: 97 pgs: 6 active+undersized+degraded, 8 stale+active+clean, 83 active+clean; 327 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 3.9 MiB/s wr, 362 op/s; 40/1608 objects degraded (2.488%) 2024-03-26T02:04:47.883 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:47 smithi019 ceph-mon[66359]: Health check failed: Degraded data redundancy: 40/1608 objects degraded (2.488%), 6 pgs degraded (PG_DEGRADED) 2024-03-26T02:04:48.128 INFO:teuthology.orchestra.run.smithi019.stdout: 5 282934 9.99 MB/sec execute 561 sec latency 43.037 ms 2024-03-26T02:04:48.161 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:47 smithi069 ceph-mon[53087]: pgmap v267: 97 pgs: 6 active+undersized+degraded, 8 stale+active+clean, 83 active+clean; 327 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 3.9 MiB/s wr, 362 op/s; 40/1608 objects degraded (2.488%) 2024-03-26T02:04:48.161 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:47 smithi069 ceph-mon[53087]: Health check failed: Degraded data redundancy: 40/1608 objects degraded (2.488%), 6 pgs degraded (PG_DEGRADED) 2024-03-26T02:04:48.161 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:47 smithi069 podman[62211]: 2024-03-26 02:04:47.863480877 +0000 UTC m=+0.627470299 container remove 444bbb4f535c346b59a45a52f5323f3e734a200b86d2f57c4373a75f4af592d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.license=GPLv2) 2024-03-26T02:04:48.161 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:47 smithi069 systemd[1]: ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.6.service: Deactivated successfully. 2024-03-26T02:04:48.161 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:47 smithi069 systemd[1]: Stopped Ceph osd.6 for 692607c6-eb12-11ee-95d0-87774f69a715. 2024-03-26T02:04:48.161 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:47 smithi069 systemd[1]: ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.6.service: Consumed 5min 19.805s CPU time. 2024-03-26T02:04:48.485 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:48 smithi069 systemd[1]: Starting Ceph osd.6 for 692607c6-eb12-11ee-95d0-87774f69a715... 2024-03-26T02:04:48.485 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:48 smithi069 podman[62311]: 2024-03-26 02:04:48.286412129 +0000 UTC m=+0.018784252 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:04:48.832 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:48 smithi069 podman[62311]: 2024-03-26 02:04:48.485165999 +0000 UTC m=+0.217538128 container create cc45353b215230fc6694ad6ad1337d989a45b62845fe6de0737d83d1ff1b88d1 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6-activate, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-26T02:04:49.109 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:48 smithi069 podman[62311]: 2024-03-26 02:04:48.967701761 +0000 UTC m=+0.700073883 container init cc45353b215230fc6694ad6ad1337d989a45b62845fe6de0737d83d1ff1b88d1 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6-activate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-26T02:04:49.109 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:48 smithi069 podman[62311]: 2024-03-26 02:04:48.971672683 +0000 UTC m=+0.704044811 container start cc45353b215230fc6694ad6ad1337d989a45b62845fe6de0737d83d1ff1b88d1 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6-activate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-26T02:04:49.109 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:49 smithi069 podman[62311]: 2024-03-26 02:04:49.067845241 +0000 UTC m=+0.800217382 container attach cc45353b215230fc6694ad6ad1337d989a45b62845fe6de0737d83d1ff1b88d1 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6-activate, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-26T02:04:49.128 INFO:teuthology.orchestra.run.smithi019.stdout: 5 283203 9.98 MB/sec execute 562 sec latency 52.155 ms 2024-03-26T02:04:49.806 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:49 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6-activate[62325]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-26T02:04:49.806 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:49 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6-activate[62325]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-26T02:04:49.806 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:49 smithi069 bash[62311]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-26T02:04:49.806 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:49 smithi069 bash[62311]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-26T02:04:49.806 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:49 smithi069 bash[62311]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-26T02:04:49.806 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:49 smithi069 bash[62311]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-26T02:04:49.806 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:49 smithi069 bash[62311]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-03-26T02:04:49.806 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:49 smithi069 bash[62311]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-26T02:04:49.806 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:49 smithi069 bash[62311]: --> ceph-volume raw activate successful for osd ID: 6 2024-03-26T02:04:49.806 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:49 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6-activate[62325]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-26T02:04:49.807 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:49 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6-activate[62325]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-26T02:04:49.807 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:49 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6-activate[62325]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-03-26T02:04:49.807 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:49 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6-activate[62325]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-26T02:04:49.807 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:49 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6-activate[62325]: --> ceph-volume raw activate successful for osd ID: 6 2024-03-26T02:04:49.807 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:49 smithi069 podman[62311]: 2024-03-26 02:04:49.484449476 +0000 UTC m=+1.216821611 container died cc45353b215230fc6694ad6ad1337d989a45b62845fe6de0737d83d1ff1b88d1 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6-activate, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-26T02:04:49.922 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:49 smithi019 ceph-mon[66359]: pgmap v268: 97 pgs: 7 active+undersized, 18 active+undersized+degraded, 2 stale+active+clean, 70 active+clean; 334 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 5.6 MiB/s wr, 568 op/s; 140/1638 objects degraded (8.547%) 2024-03-26T02:04:50.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:49 smithi069 ceph-mon[53087]: pgmap v268: 97 pgs: 7 active+undersized, 18 active+undersized+degraded, 2 stale+active+clean, 70 active+clean; 334 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 5.6 MiB/s wr, 568 op/s; 140/1638 objects degraded (8.547%) 2024-03-26T02:04:50.128 INFO:teuthology.orchestra.run.smithi019.stdout: 5 283467 9.98 MB/sec execute 563 sec latency 42.441 ms 2024-03-26T02:04:50.441 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:50 smithi069 podman[62441]: 2024-03-26 02:04:50.190262189 +0000 UTC m=+0.699642354 container remove cc45353b215230fc6694ad6ad1337d989a45b62845fe6de0737d83d1ff1b88d1 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6-activate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:04:50.441 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:50 smithi069 podman[62502]: 2024-03-26 02:04:50.313038607 +0000 UTC m=+0.021774492 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:04:50.441 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:50 smithi069 podman[62502]: 2024-03-26 02:04:50.441249467 +0000 UTC m=+0.149985375 container create 924e2a35053c1da0422a9d75ce49528904f8e881b1577fa463a94ea57172ebdb (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:04:50.702 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:50 smithi069 podman[62502]: 2024-03-26 02:04:50.677237875 +0000 UTC m=+0.385973748 container init 924e2a35053c1da0422a9d75ce49528904f8e881b1577fa463a94ea57172ebdb (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0) 2024-03-26T02:04:50.702 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:50 smithi069 podman[62502]: 2024-03-26 02:04:50.680396237 +0000 UTC m=+0.389132100 container start 924e2a35053c1da0422a9d75ce49528904f8e881b1577fa463a94ea57172ebdb (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, ceph=True, org.label-schema.schema-version=1.0) 2024-03-26T02:04:51.109 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:50 smithi069 ceph-osd[62520]: -- 172.21.15.69:0/344822942 <== mon.1 v2:172.21.15.69:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563c634d31e0 con 0x563c634a5000 2024-03-26T02:04:51.109 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:50 smithi069 bash[62502]: 924e2a35053c1da0422a9d75ce49528904f8e881b1577fa463a94ea57172ebdb 2024-03-26T02:04:51.109 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:50 smithi069 systemd[1]: Started Ceph osd.6 for 692607c6-eb12-11ee-95d0-87774f69a715. 2024-03-26T02:04:51.129 INFO:teuthology.orchestra.run.smithi019.stdout: 5 283761 9.97 MB/sec execute 564 sec latency 46.323 ms 2024-03-26T02:04:51.609 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:51 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:04:51.225+0000 7f937a6ba7c0 -1 Falling back to public interface 2024-03-26T02:04:51.942 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:51 smithi019 ceph-mon[66359]: pgmap v269: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 339 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.5 MiB/s wr, 459 op/s; 199/1656 objects degraded (12.017%) 2024-03-26T02:04:51.943 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:51 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:51.943 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:51 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:51.943 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:51 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:04:52.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:51 smithi069 ceph-mon[53087]: pgmap v269: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 339 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.5 MiB/s wr, 459 op/s; 199/1656 objects degraded (12.017%) 2024-03-26T02:04:52.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:51 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:52.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:51 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:52.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:51 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:04:52.129 INFO:teuthology.orchestra.run.smithi019.stdout: 5 284051 9.96 MB/sec execute 565 sec latency 74.122 ms 2024-03-26T02:04:52.972 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:52 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 196/1668 objects degraded (11.751%), 25 pgs degraded (PG_DEGRADED) 2024-03-26T02:04:53.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:52 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 196/1668 objects degraded (11.751%), 25 pgs degraded (PG_DEGRADED) 2024-03-26T02:04:53.128 INFO:teuthology.orchestra.run.smithi019.stdout: 5 284353 9.96 MB/sec execute 566 sec latency 59.592 ms 2024-03-26T02:04:53.996 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:53 smithi019 ceph-mon[66359]: pgmap v270: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 334 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 5.2 MiB/s wr, 520 op/s; 196/1668 objects degraded (11.751%) 2024-03-26T02:04:54.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:53 smithi069 ceph-mon[53087]: pgmap v270: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 334 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 5.2 MiB/s wr, 520 op/s; 196/1668 objects degraded (11.751%) 2024-03-26T02:04:54.129 INFO:teuthology.orchestra.run.smithi019.stdout: 5 284665 9.95 MB/sec execute 567 sec latency 67.122 ms 2024-03-26T02:04:55.129 INFO:teuthology.orchestra.run.smithi019.stdout: 5 285018 9.95 MB/sec execute 568 sec latency 75.126 ms 2024-03-26T02:04:56.057 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:55 smithi019 ceph-mon[66359]: pgmap v271: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 332 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.8 MiB/s wr, 452 op/s; 192/1653 objects degraded (11.615%) 2024-03-26T02:04:56.058 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:55 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:56.058 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:55 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:56.129 INFO:teuthology.orchestra.run.smithi019.stdout: 5 285402 9.95 MB/sec execute 569 sec latency 51.272 ms 2024-03-26T02:04:56.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:55 smithi069 ceph-mon[53087]: pgmap v271: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 332 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.8 MiB/s wr, 452 op/s; 192/1653 objects degraded (11.615%) 2024-03-26T02:04:56.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:55 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:56.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:55 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:57.129 INFO:teuthology.orchestra.run.smithi019.stdout: 5 285722 9.95 MB/sec execute 570 sec latency 58.799 ms 2024-03-26T02:04:57.818 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:57 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:57.818 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:57 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:57.819 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:57 smithi069 ceph-mon[53087]: pgmap v272: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 326 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.9 MiB/s wr, 480 op/s; 193/1623 objects degraded (11.892%) 2024-03-26T02:04:57.851 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:57 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:57.851 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:57 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:04:57.852 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:57 smithi019 ceph-mon[66359]: pgmap v272: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 326 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.9 MiB/s wr, 480 op/s; 193/1623 objects degraded (11.892%) 2024-03-26T02:04:58.129 INFO:teuthology.orchestra.run.smithi019.stdout: 5 286048 9.95 MB/sec execute 571 sec latency 52.182 ms 2024-03-26T02:04:58.886 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:58 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 199/1629 objects degraded (12.216%), 25 pgs degraded (PG_DEGRADED) 2024-03-26T02:04:59.129 INFO:teuthology.orchestra.run.smithi019.stdout: 5 286362 9.95 MB/sec execute 572 sec latency 59.969 ms 2024-03-26T02:04:59.152 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:58 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 199/1629 objects degraded (12.216%), 25 pgs degraded (PG_DEGRADED) 2024-03-26T02:04:59.609 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:04:59 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:04:59.261+0000 7f937a6ba7c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-03-26T02:04:59.906 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:04:59 smithi019 ceph-mon[66359]: pgmap v273: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 329 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.3 MiB/s wr, 567 op/s; 199/1629 objects degraded (12.216%) 2024-03-26T02:05:00.129 INFO:teuthology.orchestra.run.smithi019.stdout: 5 286626 9.95 MB/sec execute 573 sec latency 85.641 ms 2024-03-26T02:05:00.232 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:04:59 smithi069 ceph-mon[53087]: pgmap v273: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 329 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.3 MiB/s wr, 567 op/s; 199/1629 objects degraded (12.216%) 2024-03-26T02:05:00.938 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:00 smithi019 ceph-mon[66359]: from='osd.6 [v2:172.21.15.69:6816/1361710868,v1:172.21.15.69:6817/1361710868]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T02:05:00.938 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:00 smithi019 ceph-mon[66359]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T02:05:00.938 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:00 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:00.938 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:00 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:00.938 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:00 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:05:00.938 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:00 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:05:00.938 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:00 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:00.938 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:00 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:05:00.938 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:00 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:00.939 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:00 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:01.109 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:05:00 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:05:00.729+0000 7f937a6ba7c0 -1 osd.6 147 log_to_monitors true 2024-03-26T02:05:01.109 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:05:00 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:05:00.901+0000 7f936bb12700 -1 osd.6 147 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:05:01.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:00 smithi069 ceph-mon[53087]: from='osd.6 [v2:172.21.15.69:6816/1361710868,v1:172.21.15.69:6817/1361710868]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T02:05:01.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:00 smithi069 ceph-mon[53087]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T02:05:01.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:00 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:01.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:00 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:01.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:00 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:05:01.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:00 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:05:01.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:00 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:01.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:00 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:05:01.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:00 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:01.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:00 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:01.129 INFO:teuthology.orchestra.run.smithi019.stdout: 5 286948 9.95 MB/sec execute 574 sec latency 46.663 ms 2024-03-26T02:05:02.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:01 smithi019 ceph-mon[66359]: pgmap v274: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 325 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 4.1 MiB/s wr, 427 op/s; 196/1638 objects degraded (11.966%) 2024-03-26T02:05:02.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:01 smithi019 ceph-mon[66359]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T02:05:02.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:01 smithi019 ceph-mon[66359]: osdmap e150: 8 total, 7 up, 8 in 2024-03-26T02:05:02.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:01 smithi019 ceph-mon[66359]: from='osd.6 [v2:172.21.15.69:6816/1361710868,v1:172.21.15.69:6817/1361710868]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-03-26T02:05:02.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:01 smithi019 ceph-mon[66359]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-03-26T02:05:02.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:01 smithi019 ceph-mon[66359]: failed to encode map e150 with expected crc 2024-03-26T02:05:02.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:05:02.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:05:02.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:05:02.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:05:02.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-26T02:05:02.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:01 smithi019 ceph-mon[66359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-26T02:05:02.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:01 smithi019 ceph-mon[66359]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-26T02:05:02.129 INFO:teuthology.orchestra.run.smithi019.stdout: 5 287182 9.94 MB/sec execute 575 sec latency 264.910 ms 2024-03-26T02:05:02.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:01 smithi069 ceph-mon[53087]: pgmap v274: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 325 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 4.1 MiB/s wr, 427 op/s; 196/1638 objects degraded (11.966%) 2024-03-26T02:05:02.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:01 smithi069 ceph-mon[53087]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T02:05:02.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:01 smithi069 ceph-mon[53087]: osdmap e150: 8 total, 7 up, 8 in 2024-03-26T02:05:02.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:01 smithi069 ceph-mon[53087]: from='osd.6 [v2:172.21.15.69:6816/1361710868,v1:172.21.15.69:6817/1361710868]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-03-26T02:05:02.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:01 smithi069 ceph-mon[53087]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-03-26T02:05:02.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:01 smithi069 ceph-mon[53087]: failed to encode map e150 with expected crc 2024-03-26T02:05:02.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:05:02.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:05:02.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:05:02.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:05:02.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-26T02:05:02.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:01 smithi069 ceph-mon[53087]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-26T02:05:02.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:01 smithi069 ceph-mon[53087]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-26T02:05:03.129 INFO:teuthology.orchestra.run.smithi019.stdout: 5 287182 9.92 MB/sec execute 576 sec latency 1264.936 ms 2024-03-26T02:05:03.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:02 smithi019 ceph-mon[66359]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:05:03.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:02 smithi019 ceph-mon[66359]: failed to encode map e151 with expected crc 2024-03-26T02:05:03.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:02 smithi019 ceph-mon[66359]: osd.6 [v2:172.21.15.69:6816/1361710868,v1:172.21.15.69:6817/1361710868] boot 2024-03-26T02:05:03.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:02 smithi019 ceph-mon[66359]: osdmap e151: 8 total, 8 up, 8 in 2024-03-26T02:05:03.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:02 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:05:03.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:02 smithi019 ceph-mon[66359]: failed to encode map e151 with expected crc 2024-03-26T02:05:03.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:02 smithi019 ceph-mon[66359]: osdmap e152: 8 total, 8 up, 8 in 2024-03-26T02:05:03.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:02 smithi069 ceph-mon[53087]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:05:03.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:02 smithi069 ceph-mon[53087]: failed to encode map e151 with expected crc 2024-03-26T02:05:03.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:02 smithi069 ceph-mon[53087]: osd.6 [v2:172.21.15.69:6816/1361710868,v1:172.21.15.69:6817/1361710868] boot 2024-03-26T02:05:03.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:02 smithi069 ceph-mon[53087]: osdmap e151: 8 total, 8 up, 8 in 2024-03-26T02:05:03.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:02 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:05:03.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:02 smithi069 ceph-mon[53087]: failed to encode map e151 with expected crc 2024-03-26T02:05:03.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:02 smithi069 ceph-mon[53087]: osdmap e152: 8 total, 8 up, 8 in 2024-03-26T02:05:04.129 INFO:teuthology.orchestra.run.smithi019.stdout: 5 287182 9.91 MB/sec execute 577 sec latency 2264.983 ms 2024-03-26T02:05:04.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:03 smithi019 ceph-mon[66359]: pgmap v277: 97 pgs: 7 peering, 5 active+undersized, 23 active+undersized+degraded, 62 active+clean; 331 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 4.5 MiB/s wr, 398 op/s; 180/1725 objects degraded (10.435%) 2024-03-26T02:05:04.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:03 smithi019 ceph-mon[66359]: failed to encode map e152 with expected crc 2024-03-26T02:05:04.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:03 smithi019 ceph-mon[66359]: failed to encode map e152 with expected crc 2024-03-26T02:05:04.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:03 smithi019 ceph-mon[66359]: failed to encode map e152 with expected crc 2024-03-26T02:05:04.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:03 smithi019 ceph-mon[66359]: osdmap e153: 8 total, 8 up, 8 in 2024-03-26T02:05:04.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:03 smithi069 ceph-mon[53087]: pgmap v277: 97 pgs: 7 peering, 5 active+undersized, 23 active+undersized+degraded, 62 active+clean; 331 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 4.5 MiB/s wr, 398 op/s; 180/1725 objects degraded (10.435%) 2024-03-26T02:05:04.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:03 smithi069 ceph-mon[53087]: failed to encode map e152 with expected crc 2024-03-26T02:05:04.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:03 smithi069 ceph-mon[53087]: failed to encode map e152 with expected crc 2024-03-26T02:05:04.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:03 smithi069 ceph-mon[53087]: failed to encode map e152 with expected crc 2024-03-26T02:05:04.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:03 smithi069 ceph-mon[53087]: osdmap e153: 8 total, 8 up, 8 in 2024-03-26T02:05:05.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:04 smithi019 ceph-mon[66359]: failed to encode map e153 with expected crc 2024-03-26T02:05:05.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:04 smithi019 ceph-mon[66359]: failed to encode map e153 with expected crc 2024-03-26T02:05:05.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:04 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 180/1725 objects degraded (10.435%), 23 pgs degraded (PG_DEGRADED) 2024-03-26T02:05:05.129 INFO:teuthology.orchestra.run.smithi019.stdout: 5 287294 9.90 MB/sec execute 578 sec latency 2657.756 ms 2024-03-26T02:05:05.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:04 smithi069 ceph-mon[53087]: failed to encode map e153 with expected crc 2024-03-26T02:05:05.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:04 smithi069 ceph-mon[53087]: failed to encode map e153 with expected crc 2024-03-26T02:05:05.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:04 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 180/1725 objects degraded (10.435%), 23 pgs degraded (PG_DEGRADED) 2024-03-26T02:05:06.129 INFO:teuthology.orchestra.run.smithi019.stdout: 5 287371 9.88 MB/sec execute 579 sec latency 1029.805 ms 2024-03-26T02:05:06.162 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:05 smithi019 ceph-mon[66359]: pgmap v280: 97 pgs: 1 active+recovery_wait+degraded, 2 remapped+peering, 7 peering, 1 active+undersized, 13 active+undersized+degraded, 73 active+clean; 329 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 5.0 MiB/s wr, 293 op/s; 104/1671 objects degraded (6.224%); 314 KiB/s, 71 keys/s, 14 objects/s recovering 2024-03-26T02:05:06.162 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:05 smithi019 ceph-mon[66359]: failed to encode map e154 with expected crc 2024-03-26T02:05:06.162 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:05 smithi019 ceph-mon[66359]: osdmap e154: 8 total, 8 up, 8 in 2024-03-26T02:05:06.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:05 smithi069 ceph-mon[53087]: pgmap v280: 97 pgs: 1 active+recovery_wait+degraded, 2 remapped+peering, 7 peering, 1 active+undersized, 13 active+undersized+degraded, 73 active+clean; 329 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 5.0 MiB/s wr, 293 op/s; 104/1671 objects degraded (6.224%); 314 KiB/s, 71 keys/s, 14 objects/s recovering 2024-03-26T02:05:06.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:05 smithi069 ceph-mon[53087]: failed to encode map e154 with expected crc 2024-03-26T02:05:06.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:05 smithi069 ceph-mon[53087]: osdmap e154: 8 total, 8 up, 8 in 2024-03-26T02:05:06.927 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:06 smithi019 ceph-mon[66359]: osdmap e155: 8 total, 8 up, 8 in 2024-03-26T02:05:06.927 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:06 smithi019 ceph-mon[66359]: failed to encode map e155 with expected crc 2024-03-26T02:05:06.927 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:06 smithi019 ceph-mon[66359]: failed to encode map e155 with expected crc 2024-03-26T02:05:06.927 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:06 smithi019 ceph-mon[66359]: failed to encode map e155 with expected crc 2024-03-26T02:05:06.927 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:06 smithi019 ceph-mon[66359]: failed to encode map e155 with expected crc 2024-03-26T02:05:07.129 INFO:teuthology.orchestra.run.smithi019.stdout: 5 287718 9.88 MB/sec execute 580 sec latency 392.870 ms 2024-03-26T02:05:07.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:06 smithi069 ceph-mon[53087]: osdmap e155: 8 total, 8 up, 8 in 2024-03-26T02:05:07.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:06 smithi069 ceph-mon[53087]: failed to encode map e155 with expected crc 2024-03-26T02:05:07.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:06 smithi069 ceph-mon[53087]: failed to encode map e155 with expected crc 2024-03-26T02:05:07.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:06 smithi069 ceph-mon[53087]: failed to encode map e155 with expected crc 2024-03-26T02:05:07.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:06 smithi069 ceph-mon[53087]: failed to encode map e155 with expected crc 2024-03-26T02:05:08.129 INFO:teuthology.orchestra.run.smithi019.stdout: 5 287868 9.87 MB/sec execute 581 sec latency 822.186 ms 2024-03-26T02:05:08.193 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:07 smithi019 ceph-mon[66359]: pgmap v283: 97 pgs: 1 active+recovery_wait+degraded, 2 remapped+peering, 9 peering, 85 active+clean; 330 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 3.3 MiB/s wr, 441 op/s; 16/1623 objects degraded (0.986%); 986 KiB/s, 71 keys/s, 37 objects/s recovering 2024-03-26T02:05:08.193 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:07 smithi019 ceph-mon[66359]: failed to encode map e156 with expected crc 2024-03-26T02:05:08.193 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:07 smithi019 ceph-mon[66359]: osdmap e156: 8 total, 8 up, 8 in 2024-03-26T02:05:08.193 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:07 smithi019 ceph-mon[66359]: failed to encode map e156 with expected crc 2024-03-26T02:05:08.193 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:07 smithi019 ceph-mon[66359]: failed to encode map e156 with expected crc 2024-03-26T02:05:08.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:07 smithi069 ceph-mon[53087]: pgmap v283: 97 pgs: 1 active+recovery_wait+degraded, 2 remapped+peering, 9 peering, 85 active+clean; 330 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 3.3 MiB/s wr, 441 op/s; 16/1623 objects degraded (0.986%); 986 KiB/s, 71 keys/s, 37 objects/s recovering 2024-03-26T02:05:08.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:07 smithi069 ceph-mon[53087]: failed to encode map e156 with expected crc 2024-03-26T02:05:08.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:07 smithi069 ceph-mon[53087]: osdmap e156: 8 total, 8 up, 8 in 2024-03-26T02:05:08.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:07 smithi069 ceph-mon[53087]: failed to encode map e156 with expected crc 2024-03-26T02:05:08.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:07 smithi069 ceph-mon[53087]: failed to encode map e156 with expected crc 2024-03-26T02:05:09.129 INFO:teuthology.orchestra.run.smithi019.stdout: 5 288234 9.87 MB/sec execute 582 sec latency 54.863 ms 2024-03-26T02:05:09.203 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:08 smithi019 ceph-mon[66359]: osdmap e157: 8 total, 8 up, 8 in 2024-03-26T02:05:09.204 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:08 smithi019 ceph-mon[66359]: failed to encode map e157 with expected crc 2024-03-26T02:05:09.259 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:08 smithi069 ceph-mon[53087]: osdmap e157: 8 total, 8 up, 8 in 2024-03-26T02:05:09.260 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:08 smithi069 ceph-mon[53087]: failed to encode map e157 with expected crc 2024-03-26T02:05:09.982 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:09 smithi019 ceph-mon[66359]: pgmap v286: 97 pgs: 2 peering, 95 active+clean; 330 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.0 KiB/s rd, 2.5 MiB/s wr, 389 op/s; 1.9 MiB/s, 41 objects/s recovering 2024-03-26T02:05:09.982 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:09 smithi019 ceph-mon[66359]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 16/1623 objects degraded (0.986%), 1 pg degraded) 2024-03-26T02:05:09.982 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:09 smithi019 ceph-mon[66359]: Cluster is now healthy 2024-03-26T02:05:10.129 INFO:teuthology.orchestra.run.smithi019.stdout: 5 288589 9.86 MB/sec execute 583 sec latency 48.858 ms 2024-03-26T02:05:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:09 smithi069 ceph-mon[53087]: pgmap v286: 97 pgs: 2 peering, 95 active+clean; 330 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.0 KiB/s rd, 2.5 MiB/s wr, 389 op/s; 1.9 MiB/s, 41 objects/s recovering 2024-03-26T02:05:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:09 smithi069 ceph-mon[53087]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 16/1623 objects degraded (0.986%), 1 pg degraded) 2024-03-26T02:05:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:09 smithi069 ceph-mon[53087]: Cluster is now healthy 2024-03-26T02:05:11.129 INFO:teuthology.orchestra.run.smithi019.stdout: 5 288969 9.86 MB/sec execute 584 sec latency 42.090 ms 2024-03-26T02:05:11.573 INFO:teuthology.orchestra.run.smithi019.stdout:true 2024-03-26T02:05:12.008 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:11 smithi019 ceph-mon[66359]: pgmap v287: 97 pgs: 2 peering, 95 active+clean; 330 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 2.0 MiB/s wr, 289 op/s; 1.3 MiB/s, 28 objects/s recovering 2024-03-26T02:05:12.129 INFO:teuthology.orchestra.run.smithi019.stdout: 5 289246 9.86 MB/sec execute 585 sec latency 64.427 ms 2024-03-26T02:05:12.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:11 smithi069 ceph-mon[53087]: pgmap v287: 97 pgs: 2 peering, 95 active+clean; 330 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 2.0 MiB/s wr, 289 op/s; 1.3 MiB/s, 28 objects/s recovering 2024-03-26T02:05:12.622 INFO:teuthology.orchestra.run.smithi019.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:05:12.622 INFO:teuthology.orchestra.run.smithi019.stdout:alertmanager.smithi019 smithi019 *:9093,9094 running (9m) 2m ago 18m 14.5M - 0.23.0 ba2b418f427c a6630841c2b2 2024-03-26T02:05:12.623 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi019 smithi019 running (5m) 2m ago 18m 6660k - 19.0.0-1775-g5e70124b 5f21c9264136 c5f91a47ddf2 2024-03-26T02:05:12.623 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi069 smithi069 running (5m) 17s ago 17m 6669k - 19.0.0-1775-g5e70124b 5f21c9264136 05128803b49e 2024-03-26T02:05:12.623 INFO:teuthology.orchestra.run.smithi019.stdout:grafana.smithi019 smithi019 *:3000 running (8m) 2m ago 18m 46.9M - 8.3.5 dad864ee21e9 a97dea3b5fec 2024-03-26T02:05:12.623 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi019.lmryom smithi019 running (12m) 2m ago 12m 38.4M - 17.2.0 e1d6a67b021e 91a7c2942715 2024-03-26T02:05:12.623 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi069.spgwgo smithi069 running (12m) 17s ago 12m 15.4M - 17.2.0 e1d6a67b021e 94d96338dbe1 2024-03-26T02:05:12.623 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi019.ixuvdm smithi019 *:8443,9283,8765 running (8m) 2m ago 19m 508M - 19.0.0-1775-g5e70124b 5f21c9264136 4460b22c7a56 2024-03-26T02:05:12.623 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi069.wpbewu smithi069 *:8443,9283,8765 running (7m) 17s ago 17m 454M - 19.0.0-1775-g5e70124b 5f21c9264136 380fd33842a8 2024-03-26T02:05:12.623 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi019 smithi019 running (6m) 2m ago 19m 52.0M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 c18934705774 2024-03-26T02:05:12.623 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi069 smithi069 running (6m) 17s ago 17m 43.2M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 a856bd263104 2024-03-26T02:05:12.623 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.0.0.smithi019.qbizxo smithi019 *:2049 running (11m) 2m ago 11m 283M - 4.0 e1d6a67b021e cd6628a9075d 2024-03-26T02:05:12.623 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.1.0.smithi069.uawdhc smithi069 *:2049 running (11m) 17s ago 11m 85.8M - 4.0 e1d6a67b021e 6ee3695e7a10 2024-03-26T02:05:12.624 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi019 smithi019 *:9100 running (8m) 2m ago 18m 10.2M - 1.3.1 1dbe0e931976 0d1d184bec63 2024-03-26T02:05:12.624 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi069 smithi069 *:9100 running (8m) 17s ago 17m 22.6M - 1.3.1 1dbe0e931976 c49a006f94e7 2024-03-26T02:05:12.624 INFO:teuthology.orchestra.run.smithi019.stdout:osd.0 smithi019 running (4m) 2m ago 16m 340M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 3f931cfe320a 2024-03-26T02:05:12.624 INFO:teuthology.orchestra.run.smithi019.stdout:osd.1 smithi019 running (3m) 2m ago 16m 289M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 758caf4ed841 2024-03-26T02:05:12.624 INFO:teuthology.orchestra.run.smithi019.stdout:osd.2 smithi019 running (3m) 2m ago 15m 228M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 6a9b701517ca 2024-03-26T02:05:12.624 INFO:teuthology.orchestra.run.smithi019.stdout:osd.3 smithi019 running (2m) 2m ago 15m 12.6M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 0401940d882b 2024-03-26T02:05:12.624 INFO:teuthology.orchestra.run.smithi019.stdout:osd.4 smithi069 running (96s) 17s ago 15m 293M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 6eb3b2ce9978 2024-03-26T02:05:12.624 INFO:teuthology.orchestra.run.smithi019.stdout:osd.5 smithi069 running (59s) 17s ago 14m 265M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 fb31f0e68fb7 2024-03-26T02:05:12.624 INFO:teuthology.orchestra.run.smithi019.stdout:osd.6 smithi069 running (22s) 17s ago 14m 11.9M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 924e2a35053c 2024-03-26T02:05:12.624 INFO:teuthology.orchestra.run.smithi019.stdout:osd.7 smithi069 running (13m) 17s ago 13m 683M 2704M 17.2.0 e1d6a67b021e d175fbc9ff8a 2024-03-26T02:05:12.624 INFO:teuthology.orchestra.run.smithi019.stdout:prometheus.smithi019 smithi019 *:9095 running (7m) 2m ago 17m 67.6M - 2.33.4 514e6a882f6e 29f89db72a07 2024-03-26T02:05:13.041 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:12 smithi019 ceph-mon[66359]: from='client.34428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:05:13.041 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:12 smithi019 ceph-mon[66359]: from='client.34432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:05:13.129 INFO:teuthology.orchestra.run.smithi019.stdout: 5 289585 9.87 MB/sec execute 586 sec latency 66.164 ms 2024-03-26T02:05:13.250 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:05:13.250 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": { 2024-03-26T02:05:13.250 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:05:13.250 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:05:13.250 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": { 2024-03-26T02:05:13.250 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:05:13.250 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:05:13.251 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": { 2024-03-26T02:05:13.251 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-26T02:05:13.251 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 7 2024-03-26T02:05:13.251 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:05:13.251 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": { 2024-03-26T02:05:13.251 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:05:13.251 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:05:13.251 INFO:teuthology.orchestra.run.smithi019.stdout: "overall": { 2024-03-26T02:05:13.251 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-03-26T02:05:13.251 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 11 2024-03-26T02:05:13.252 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-26T02:05:13.252 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:05:13.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:12 smithi069 ceph-mon[53087]: from='client.34428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:05:13.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:12 smithi069 ceph-mon[53087]: from='client.34432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:05:13.797 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:05:13.797 INFO:teuthology.orchestra.run.smithi019.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2", 2024-03-26T02:05:13.798 INFO:teuthology.orchestra.run.smithi019.stdout: "in_progress": true, 2024-03-26T02:05:13.798 INFO:teuthology.orchestra.run.smithi019.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T02:05:13.798 INFO:teuthology.orchestra.run.smithi019.stdout: "services_complete": [ 2024-03-26T02:05:13.798 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr", 2024-03-26T02:05:13.798 INFO:teuthology.orchestra.run.smithi019.stdout: "crash", 2024-03-26T02:05:13.798 INFO:teuthology.orchestra.run.smithi019.stdout: "mon" 2024-03-26T02:05:13.798 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-26T02:05:13.798 INFO:teuthology.orchestra.run.smithi019.stdout: "progress": "13/23 daemons upgraded", 2024-03-26T02:05:13.798 INFO:teuthology.orchestra.run.smithi019.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T02:05:13.798 INFO:teuthology.orchestra.run.smithi019.stdout: "is_paused": false 2024-03-26T02:05:13.798 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:05:14.061 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:13 smithi019 ceph-mon[66359]: from='client.34436 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:05:14.061 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:13 smithi019 ceph-mon[66359]: pgmap v288: 97 pgs: 97 active+clean; 328 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 3.0 MiB/s wr, 348 op/s; 1.9 MiB/s, 17 objects/s recovering 2024-03-26T02:05:14.061 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:13 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/238517521' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:05:14.129 INFO:teuthology.orchestra.run.smithi019.stdout: 5 289907 9.86 MB/sec execute 587 sec latency 53.056 ms 2024-03-26T02:05:14.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:13 smithi069 ceph-mon[53087]: from='client.34436 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:05:14.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:13 smithi069 ceph-mon[53087]: pgmap v288: 97 pgs: 97 active+clean; 328 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 3.0 MiB/s wr, 348 op/s; 1.9 MiB/s, 17 objects/s recovering 2024-03-26T02:05:14.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:13 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/238517521' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:05:14.502 INFO:teuthology.orchestra.run.smithi019.stdout:HEALTH_OK 2024-03-26T02:05:15.070 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:14 smithi019 ceph-mon[66359]: from='client.34444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:05:15.070 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:14 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/3077873705' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:05:15.130 INFO:teuthology.orchestra.run.smithi019.stdout: 5 290277 9.86 MB/sec execute 588 sec latency 35.488 ms 2024-03-26T02:05:15.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:14 smithi069 ceph-mon[53087]: from='client.34444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:05:15.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:14 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/3077873705' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:05:16.081 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:15 smithi019 ceph-mon[66359]: pgmap v289: 97 pgs: 97 active+clean; 329 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 3.7 MiB/s wr, 421 op/s; 1.6 MiB/s, 9 objects/s recovering 2024-03-26T02:05:16.081 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:15 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:16.081 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:15 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:05:16.081 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:15 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-26T02:05:16.129 INFO:teuthology.orchestra.run.smithi019.stdout: 5 290718 9.86 MB/sec execute 589 sec latency 30.973 ms 2024-03-26T02:05:16.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:15 smithi069 ceph-mon[53087]: pgmap v289: 97 pgs: 97 active+clean; 329 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 3.7 MiB/s wr, 421 op/s; 1.6 MiB/s, 9 objects/s recovering 2024-03-26T02:05:16.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:15 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:16.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:15 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:05:16.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:15 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-26T02:05:17.063 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:16 smithi069 ceph-mon[53087]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-26T02:05:17.063 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:16 smithi069 ceph-mon[53087]: Upgrade: osd.7 is safe to restart 2024-03-26T02:05:17.088 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:16 smithi019 ceph-mon[66359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-26T02:05:17.089 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:16 smithi019 ceph-mon[66359]: Upgrade: osd.7 is safe to restart 2024-03-26T02:05:17.129 INFO:teuthology.orchestra.run.smithi019.stdout: 5 291148 9.87 MB/sec execute 590 sec latency 35.839 ms 2024-03-26T02:05:18.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:17 smithi069 ceph-mon[53087]: pgmap v290: 97 pgs: 97 active+clean; 330 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.6 MiB/s wr, 366 op/s; 1.3 MiB/s, 7 objects/s recovering 2024-03-26T02:05:18.118 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:17 smithi019 ceph-mon[66359]: pgmap v290: 97 pgs: 97 active+clean; 330 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.6 MiB/s wr, 366 op/s; 1.3 MiB/s, 7 objects/s recovering 2024-03-26T02:05:18.129 INFO:teuthology.orchestra.run.smithi019.stdout: 5 291407 9.86 MB/sec execute 591 sec latency 58.290 ms 2024-03-26T02:05:19.047 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:19 smithi069 ceph-mon[53087]: Upgrade: Updating osd.7 2024-03-26T02:05:19.047 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:19 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:19.047 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:19 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T02:05:19.048 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:19 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:05:19.048 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:19 smithi069 ceph-mon[53087]: Deploying daemon osd.7 on smithi069 2024-03-26T02:05:19.127 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:19 smithi019 ceph-mon[66359]: Upgrade: Updating osd.7 2024-03-26T02:05:19.127 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:19 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:19.127 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:19 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T02:05:19.127 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:19 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:05:19.127 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:19 smithi019 ceph-mon[66359]: Deploying daemon osd.7 on smithi069 2024-03-26T02:05:19.129 INFO:teuthology.orchestra.run.smithi019.stdout: 5 291654 9.85 MB/sec execute 592 sec latency 54.950 ms 2024-03-26T02:05:20.079 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:20 smithi069 ceph-mon[53087]: pgmap v291: 97 pgs: 97 active+clean; 334 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.5 MiB/s wr, 410 op/s; 1.1 MiB/s, 3 objects/s recovering 2024-03-26T02:05:20.129 INFO:teuthology.orchestra.run.smithi019.stdout: 5 291930 9.85 MB/sec execute 593 sec latency 65.967 ms 2024-03-26T02:05:20.139 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:20 smithi019 ceph-mon[66359]: pgmap v291: 97 pgs: 97 active+clean; 334 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.5 MiB/s wr, 410 op/s; 1.1 MiB/s, 3 objects/s recovering 2024-03-26T02:05:21.130 INFO:teuthology.orchestra.run.smithi019.stdout: 5 292252 9.84 MB/sec execute 594 sec latency 67.486 ms 2024-03-26T02:05:21.941 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:21 smithi019 ceph-mon[66359]: pgmap v292: 97 pgs: 97 active+clean; 332 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.3 KiB/s rd, 4.4 MiB/s wr, 431 op/s; 683 KiB/s, 0 objects/s recovering 2024-03-26T02:05:21.941 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:21 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:22.083 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:21 smithi069 ceph-mon[53087]: pgmap v292: 97 pgs: 97 active+clean; 332 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.3 KiB/s rd, 4.4 MiB/s wr, 431 op/s; 683 KiB/s, 0 objects/s recovering 2024-03-26T02:05:22.083 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:21 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:22.083 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:21 smithi069 systemd[1]: Stopping Ceph osd.7 for 692607c6-eb12-11ee-95d0-87774f69a715... 2024-03-26T02:05:22.130 INFO:teuthology.orchestra.run.smithi019.stdout: 5 292589 9.84 MB/sec execute 595 sec latency 45.643 ms 2024-03-26T02:05:22.359 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:22 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[40323]: 2024-03-26T02:05:22.103+0000 7f495005f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:05:22.359 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:22 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[40323]: 2024-03-26T02:05:22.103+0000 7f495005f700 -1 osd.7 157 *** Got signal Terminated *** 2024-03-26T02:05:22.359 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:22 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[40323]: 2024-03-26T02:05:22.103+0000 7f495005f700 -1 osd.7 157 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:05:22.956 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:22 smithi019 ceph-mon[66359]: osd.7 marked itself down and dead 2024-03-26T02:05:23.130 INFO:teuthology.orchestra.run.smithi019.stdout: 5 292802 9.84 MB/sec execute 596 sec latency 269.207 ms 2024-03-26T02:05:23.190 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:22 smithi069 ceph-mon[53087]: osd.7 marked itself down and dead 2024-03-26T02:05:23.859 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:23 smithi069 podman[64266]: 2024-03-26 02:05:23.581504516 +0000 UTC m=+1.595649512 container died d175fbc9ff8a481bac1c32f2e73ba1412fff8b893964ea8c4844d96aa37e1826 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, name=centos-stream, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, RELEASE=HEAD, distribution-scope=public, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-03-26T02:05:24.123 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:23 smithi019 ceph-mon[66359]: pgmap v293: 97 pgs: 97 active+clean; 333 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.4 MiB/s wr, 556 op/s; 683 KiB/s, 0 objects/s recovering 2024-03-26T02:05:24.124 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:23 smithi019 ceph-mon[66359]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:05:24.124 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:23 smithi019 ceph-mon[66359]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-26T02:05:24.124 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:23 smithi019 ceph-mon[66359]: osdmap e158: 8 total, 7 up, 8 in 2024-03-26T02:05:24.130 INFO:teuthology.orchestra.run.smithi019.stdout: 5 292878 9.82 MB/sec execute 597 sec latency 1031.277 ms 2024-03-26T02:05:24.179 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:23 smithi069 ceph-mon[53087]: pgmap v293: 97 pgs: 97 active+clean; 333 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.4 MiB/s wr, 556 op/s; 683 KiB/s, 0 objects/s recovering 2024-03-26T02:05:24.179 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:23 smithi069 ceph-mon[53087]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:05:24.179 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:23 smithi069 ceph-mon[53087]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-26T02:05:24.179 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:23 smithi069 ceph-mon[53087]: osdmap e158: 8 total, 7 up, 8 in 2024-03-26T02:05:24.546 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:24 smithi069 podman[64266]: 2024-03-26 02:05:24.30323803 +0000 UTC m=+2.317383023 container cleanup d175fbc9ff8a481bac1c32f2e73ba1412fff8b893964ea8c4844d96aa37e1826 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7, io.openshift.expose-services=, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, io.k8s.display-name=CentOS Stream 8) 2024-03-26T02:05:24.546 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:24 smithi069 bash[64266]: ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7 2024-03-26T02:05:24.859 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:24 smithi069 podman[64293]: 2024-03-26 02:05:24.546413016 +0000 UTC m=+0.962949434 container remove d175fbc9ff8a481bac1c32f2e73ba1412fff8b893964ea8c4844d96aa37e1826 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, architecture=x86_64, RELEASE=HEAD, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, io.openshift.expose-services=, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, vcs-type=git, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-03-26T02:05:25.130 INFO:teuthology.orchestra.run.smithi019.stdout: 5 293255 9.83 MB/sec execute 598 sec latency 62.483 ms 2024-03-26T02:05:25.147 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:24 smithi019 ceph-mon[66359]: osdmap e159: 8 total, 7 up, 8 in 2024-03-26T02:05:25.197 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:24 smithi069 ceph-mon[53087]: osdmap e159: 8 total, 7 up, 8 in 2024-03-26T02:05:25.197 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:25 smithi069 podman[64382]: 2024-03-26 02:05:24.948548091 +0000 UTC m=+0.022626660 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:05:25.197 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:25 smithi069 podman[64382]: 2024-03-26 02:05:25.072422173 +0000 UTC m=+0.146500769 container create 9e79e6d9bf2eae9920468859f6008d0bbee782d2e83f40f7a89df3edb9753776 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=) 2024-03-26T02:05:25.609 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:25 smithi069 podman[64382]: 2024-03-26 02:05:25.30042452 +0000 UTC m=+0.374503082 container init 9e79e6d9bf2eae9920468859f6008d0bbee782d2e83f40f7a89df3edb9753776 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True) 2024-03-26T02:05:25.609 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:25 smithi069 podman[64382]: 2024-03-26 02:05:25.304949259 +0000 UTC m=+0.379027810 container start 9e79e6d9bf2eae9920468859f6008d0bbee782d2e83f40f7a89df3edb9753776 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-26T02:05:25.609 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:25 smithi069 podman[64382]: 2024-03-26 02:05:25.391974435 +0000 UTC m=+0.466052993 container attach 9e79e6d9bf2eae9920468859f6008d0bbee782d2e83f40f7a89df3edb9753776 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-26T02:05:25.900 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:25 smithi019 ceph-mon[66359]: pgmap v296: 97 pgs: 2 active+undersized, 10 active+undersized+degraded, 8 peering, 9 stale+active+clean, 68 active+clean; 331 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 5.2 MiB/s wr, 507 op/s; 58/1632 objects degraded (3.554%) 2024-03-26T02:05:25.900 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:25 smithi019 ceph-mon[66359]: Health check failed: Degraded data redundancy: 58/1632 objects degraded (3.554%), 10 pgs degraded (PG_DEGRADED) 2024-03-26T02:05:26.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:25 smithi069 ceph-mon[53087]: pgmap v296: 97 pgs: 2 active+undersized, 10 active+undersized+degraded, 8 peering, 9 stale+active+clean, 68 active+clean; 331 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 5.2 MiB/s wr, 507 op/s; 58/1632 objects degraded (3.554%) 2024-03-26T02:05:26.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:25 smithi069 ceph-mon[53087]: Health check failed: Degraded data redundancy: 58/1632 objects degraded (3.554%), 10 pgs degraded (PG_DEGRADED) 2024-03-26T02:05:26.109 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:25 smithi069 podman[64382]: 2024-03-26 02:05:25.688268486 +0000 UTC m=+0.762347047 container died 9e79e6d9bf2eae9920468859f6008d0bbee782d2e83f40f7a89df3edb9753776 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-26T02:05:26.130 INFO:teuthology.orchestra.run.smithi019.stdout: 5 293666 9.84 MB/sec execute 599 sec latency 54.895 ms 2024-03-26T02:05:27.110 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:26 smithi069 podman[64417]: 2024-03-26 02:05:26.663743304 +0000 UTC m=+0.967543740 container remove 9e79e6d9bf2eae9920468859f6008d0bbee782d2e83f40f7a89df3edb9753776 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7-deactivate, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-26T02:05:27.110 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:26 smithi069 systemd[1]: ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.7.service: Deactivated successfully. 2024-03-26T02:05:27.110 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:26 smithi069 systemd[1]: Stopped Ceph osd.7 for 692607c6-eb12-11ee-95d0-87774f69a715. 2024-03-26T02:05:27.110 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:26 smithi069 systemd[1]: ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.7.service: Consumed 8min 764ms CPU time. 2024-03-26T02:05:27.110 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:26 smithi069 systemd[1]: Starting Ceph osd.7 for 692607c6-eb12-11ee-95d0-87774f69a715... 2024-03-26T02:05:27.130 INFO:teuthology.orchestra.run.smithi019.stdout: 5 cleanup 600 sec 2024-03-26T02:05:27.320 INFO:teuthology.orchestra.run.smithi019.stdout: 0 cleanup 600 sec 2024-03-26T02:05:27.320 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-26T02:05:27.320 INFO:teuthology.orchestra.run.smithi019.stdout: Operation Count AvgLat MaxLat 2024-03-26T02:05:27.321 INFO:teuthology.orchestra.run.smithi019.stdout: ---------------------------------------- 2024-03-26T02:05:27.321 INFO:teuthology.orchestra.run.smithi019.stdout: NTCreateX 189429 4.633 2657.752 2024-03-26T02:05:27.321 INFO:teuthology.orchestra.run.smithi019.stdout: Close 139072 0.144 7.892 2024-03-26T02:05:27.321 INFO:teuthology.orchestra.run.smithi019.stdout: Rename 8024 9.476 2298.336 2024-03-26T02:05:27.321 INFO:teuthology.orchestra.run.smithi019.stdout: Unlink 38319 6.965 2652.693 2024-03-26T02:05:27.321 INFO:teuthology.orchestra.run.smithi019.stdout: Qpathinfo 171849 1.198 1067.602 2024-03-26T02:05:27.321 INFO:teuthology.orchestra.run.smithi019.stdout: Qfileinfo 29944 0.008 0.780 2024-03-26T02:05:27.321 INFO:teuthology.orchestra.run.smithi019.stdout: Qfsinfo 31476 0.625 38.481 2024-03-26T02:05:27.321 INFO:teuthology.orchestra.run.smithi019.stdout: Sfileinfo 15457 13.214 2649.623 2024-03-26T02:05:27.321 INFO:teuthology.orchestra.run.smithi019.stdout: Find 66383 1.035 13.432 2024-03-26T02:05:27.321 INFO:teuthology.orchestra.run.smithi019.stdout: WriteX 93579 13.396 2227.081 2024-03-26T02:05:27.321 INFO:teuthology.orchestra.run.smithi019.stdout: ReadX 296851 0.010 7.916 2024-03-26T02:05:27.321 INFO:teuthology.orchestra.run.smithi019.stdout: LockX 616 0.398 2.776 2024-03-26T02:05:27.321 INFO:teuthology.orchestra.run.smithi019.stdout: UnlockX 616 0.343 2.202 2024-03-26T02:05:27.322 INFO:teuthology.orchestra.run.smithi019.stdout: Flush 13292 0.002 0.034 2024-03-26T02:05:27.322 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-26T02:05:27.322 INFO:teuthology.orchestra.run.smithi019.stdout:Throughput 9.83765 MB/sec 5 clients 5 procs max_latency=2657.756 ms 2024-03-26T02:05:27.323 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'umount /mnt/foo' 2024-03-26T02:05:27.358 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done' 2024-03-26T02:05:27.434 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:27 smithi069 podman[64541]: 2024-03-26 02:05:27.060278297 +0000 UTC m=+0.023060090 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:05:27.434 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:27 smithi069 podman[64541]: 2024-03-26 02:05:27.189916283 +0000 UTC m=+0.152698063 container create 5c136ccce2b9daa8b2275ae801271260f5378525e759916168d4fe6292e7f8fa (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7-activate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-26T02:05:27.492 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 5' 2024-03-26T02:05:27.527 INFO:teuthology.orchestra.run.smithi019.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-03-26T02:05:27.527 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-26T02:05:27.527 INFO:teuthology.orchestra.run.smithi019.stdout:Running for 5 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 1 secs 2024-03-26T02:05:27.528 INFO:teuthology.orchestra.run.smithi019.stdout:0 of 5 processes prepared for launch 0 sec 2024-03-26T02:05:27.628 INFO:teuthology.orchestra.run.smithi019.stdout:5 of 5 processes prepared for launch 0 sec 2024-03-26T02:05:27.628 INFO:teuthology.orchestra.run.smithi019.stdout:releasing clients 2024-03-26T02:05:27.859 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:27 smithi069 podman[64541]: 2024-03-26 02:05:27.434223894 +0000 UTC m=+0.397005676 container init 5c136ccce2b9daa8b2275ae801271260f5378525e759916168d4fe6292e7f8fa (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7-activate, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:05:27.859 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:27 smithi069 podman[64541]: 2024-03-26 02:05:27.437416017 +0000 UTC m=+0.400197805 container start 5c136ccce2b9daa8b2275ae801271260f5378525e759916168d4fe6292e7f8fa (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-26T02:05:27.859 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:27 smithi069 podman[64541]: 2024-03-26 02:05:27.52567736 +0000 UTC m=+0.488459141 container attach 5c136ccce2b9daa8b2275ae801271260f5378525e759916168d4fe6292e7f8fa (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7-activate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-26T02:05:27.936 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:27 smithi019 ceph-mon[66359]: pgmap v297: 97 pgs: 2 active+undersized, 16 active+undersized+degraded, 8 peering, 8 stale+active+clean, 63 active+clean; 331 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 4.1 MiB/s wr, 563 op/s; 86/1617 objects degraded (5.318%) 2024-03-26T02:05:28.266 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:27 smithi069 ceph-mon[53087]: pgmap v297: 97 pgs: 2 active+undersized, 16 active+undersized+degraded, 8 peering, 8 stale+active+clean, 63 active+clean; 331 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 4.1 MiB/s wr, 563 op/s; 86/1617 objects degraded (5.318%) 2024-03-26T02:05:28.267 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:27 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7-activate[64555]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-26T02:05:28.267 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:27 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7-activate[64555]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-26T02:05:28.267 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:27 smithi069 bash[64541]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-26T02:05:28.267 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:27 smithi069 bash[64541]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-26T02:05:28.267 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:27 smithi069 bash[64541]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-26T02:05:28.267 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:27 smithi069 bash[64541]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-26T02:05:28.267 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:27 smithi069 bash[64541]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-03-26T02:05:28.267 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:27 smithi069 bash[64541]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-26T02:05:28.267 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:27 smithi069 bash[64541]: --> ceph-volume raw activate successful for osd ID: 7 2024-03-26T02:05:28.267 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:27 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7-activate[64555]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-26T02:05:28.267 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:27 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7-activate[64555]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-26T02:05:28.267 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:27 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7-activate[64555]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-03-26T02:05:28.267 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:27 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7-activate[64555]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-26T02:05:28.268 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:27 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7-activate[64555]: --> ceph-volume raw activate successful for osd ID: 7 2024-03-26T02:05:28.268 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:27 smithi069 podman[64541]: 2024-03-26 02:05:27.997271754 +0000 UTC m=+0.960053528 container died 5c136ccce2b9daa8b2275ae801271260f5378525e759916168d4fe6292e7f8fa (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7-activate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:05:28.970 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:28 smithi069 podman[64687]: 2024-03-26 02:05:28.689055931 +0000 UTC m=+0.681844057 container remove 5c136ccce2b9daa8b2275ae801271260f5378525e759916168d4fe6292e7f8fa (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7-activate, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-26T02:05:29.223 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:28 smithi069 podman[64742]: 2024-03-26 02:05:28.875382945 +0000 UTC m=+0.034628536 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:05:29.584 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:29 smithi069 podman[64742]: 2024-03-26 02:05:29.223479267 +0000 UTC m=+0.382724850 container create d8052b0a18d6edf22af2ab6e4a52aa9f5d56b69ad64c8a2ccbffbacd804f473b (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-26T02:05:29.585 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:29 smithi069 podman[64742]: 2024-03-26 02:05:29.584467379 +0000 UTC m=+0.743712955 container init d8052b0a18d6edf22af2ab6e4a52aa9f5d56b69ad64c8a2ccbffbacd804f473b (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-26T02:05:29.628 INFO:teuthology.orchestra.run.smithi019.stdout: 5 394 23.15 MB/sec execute 1 sec latency 65.555 ms 2024-03-26T02:05:29.859 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:29 smithi069 podman[64742]: 2024-03-26 02:05:29.589458078 +0000 UTC m=+0.748703661 container start d8052b0a18d6edf22af2ab6e4a52aa9f5d56b69ad64c8a2ccbffbacd804f473b (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=) 2024-03-26T02:05:29.859 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:29 smithi069 ceph-osd[64761]: -- 172.21.15.69:0/4103720249 <== mon.1 v2:172.21.15.69:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563f2283d1e0 con 0x563f2280f000 2024-03-26T02:05:29.859 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:29 smithi069 bash[64742]: d8052b0a18d6edf22af2ab6e4a52aa9f5d56b69ad64c8a2ccbffbacd804f473b 2024-03-26T02:05:29.859 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:29 smithi069 systemd[1]: Started Ceph osd.7 for 692607c6-eb12-11ee-95d0-87774f69a715. 2024-03-26T02:05:30.009 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:29 smithi019 ceph-mon[66359]: pgmap v298: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 342 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.8 MiB/s wr, 466 op/s; 266/1692 objects degraded (15.721%) 2024-03-26T02:05:30.009 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:29 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:30.009 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:29 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:30.009 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:29 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:05:30.359 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:30 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:05:30.132+0000 7f024debe7c0 -1 Falling back to public interface 2024-03-26T02:05:30.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:29 smithi069 ceph-mon[53087]: pgmap v298: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 342 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.8 MiB/s wr, 466 op/s; 266/1692 objects degraded (15.721%) 2024-03-26T02:05:30.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:29 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:30.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:29 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:30.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:29 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:05:30.628 INFO:teuthology.orchestra.run.smithi019.stdout: 5 556 24.38 MB/sec execute 2 sec latency 38.287 ms 2024-03-26T02:05:31.628 INFO:teuthology.orchestra.run.smithi019.stdout: 5 890 20.52 MB/sec execute 3 sec latency 59.935 ms 2024-03-26T02:05:31.858 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:31 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:31.858 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:31 smithi019 ceph-mon[66359]: pgmap v299: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 325 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.1 MiB/s wr, 302 op/s; 236/1491 objects degraded (15.828%) 2024-03-26T02:05:31.858 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:31 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:05:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:31 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:31 smithi069 ceph-mon[53087]: pgmap v299: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 325 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.1 MiB/s wr, 302 op/s; 236/1491 objects degraded (15.828%) 2024-03-26T02:05:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:31 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:05:32.628 INFO:teuthology.orchestra.run.smithi019.stdout: 5 1284 17.92 MB/sec execute 4 sec latency 23.493 ms 2024-03-26T02:05:32.851 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:32 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 236/1491 objects degraded (15.828%), 35 pgs degraded (PG_DEGRADED) 2024-03-26T02:05:32.870 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:32 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 236/1491 objects degraded (15.828%), 35 pgs degraded (PG_DEGRADED) 2024-03-26T02:05:33.628 INFO:teuthology.orchestra.run.smithi019.stdout: 5 cleanup 5 sec 2024-03-26T02:05:33.767 INFO:teuthology.orchestra.run.smithi019.stdout:[1914] unlink './clients/client4/~dmtmp/PARADOX/.nfs000001000000cbf200000028' failed - Device or resource busy 2024-03-26T02:05:33.767 INFO:teuthology.orchestra.run.smithi019.stdout:[1914] unlink './clients/client4/~dmtmp/PARADOX/.nfs000001000000cc150000001d' failed - Device or resource busy 2024-03-26T02:05:33.768 INFO:teuthology.orchestra.run.smithi019.stdout:[1914] unlink './clients/client4/~dmtmp/PARADOX/.nfs000001000000ccfd00000018' failed - Device or resource busy 2024-03-26T02:05:33.768 INFO:teuthology.orchestra.run.smithi019.stdout:[1914] unlink './clients/client4/~dmtmp/PARADOX/.nfs000001000000cc330000001e' failed - Device or resource busy 2024-03-26T02:05:33.769 INFO:teuthology.orchestra.run.smithi019.stdout:[1914] unlink './clients/client4/~dmtmp/PARADOX/.nfs000001000000cc380000002b' failed - Device or resource busy 2024-03-26T02:05:33.769 INFO:teuthology.orchestra.run.smithi019.stdout:[1914] unlink './clients/client4/~dmtmp/PARADOX/.nfs000001000000cbfc0000001b' failed - Device or resource busy 2024-03-26T02:05:33.770 INFO:teuthology.orchestra.run.smithi019.stdout:[1914] unlink './clients/client4/~dmtmp/PARADOX/.nfs000001000000cc1f00000023' failed - Device or resource busy 2024-03-26T02:05:33.770 INFO:teuthology.orchestra.run.smithi019.stdout:[1914] unlink './clients/client4/~dmtmp/PARADOX/.nfs000001000000cd0a00000015' failed - Device or resource busy 2024-03-26T02:05:33.772 INFO:teuthology.orchestra.run.smithi019.stdout:[1914] unlink './clients/client4/~dmtmp/PARADOX/.nfs000001000000cc0000000029' failed - Device or resource busy 2024-03-26T02:05:33.772 INFO:teuthology.orchestra.run.smithi019.stdout:[1914] unlink './clients/client4/~dmtmp/PARADOX/.nfs000001000000cc0b00000020' failed - Device or resource busy 2024-03-26T02:05:33.791 INFO:teuthology.orchestra.run.smithi019.stdout:[1914] unlink './clients/client4/~dmtmp/PARADOX/.nfs000001000000cc3d00000024' failed - Device or resource busy 2024-03-26T02:05:33.791 INFO:teuthology.orchestra.run.smithi019.stdout:[1914] unlink './clients/client4/~dmtmp/PARADOX/.nfs000001000000cc2400000026' failed - Device or resource busy 2024-03-26T02:05:33.794 INFO:teuthology.orchestra.run.smithi019.stdout:[1914] unlink './clients/client4/~dmtmp/PARADOX/.nfs000001000000cbf800000017' failed - Device or resource busy 2024-03-26T02:05:33.884 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:33 smithi019 ceph-mon[66359]: pgmap v300: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 328 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 11 MiB/s wr, 464 op/s; 184/1311 objects degraded (14.035%) 2024-03-26T02:05:33.889 INFO:teuthology.orchestra.run.smithi019.stdout: 0 cleanup 5 sec 2024-03-26T02:05:33.889 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-26T02:05:33.889 INFO:teuthology.orchestra.run.smithi019.stdout: Operation Count AvgLat MaxLat 2024-03-26T02:05:33.889 INFO:teuthology.orchestra.run.smithi019.stdout: ---------------------------------------- 2024-03-26T02:05:33.889 INFO:teuthology.orchestra.run.smithi019.stdout: NTCreateX 1352 5.099 59.092 2024-03-26T02:05:33.890 INFO:teuthology.orchestra.run.smithi019.stdout: Close 1129 5.008 59.929 2024-03-26T02:05:33.890 INFO:teuthology.orchestra.run.smithi019.stdout: Rename 55 8.193 28.390 2024-03-26T02:05:33.890 INFO:teuthology.orchestra.run.smithi019.stdout: Unlink 120 4.418 19.871 2024-03-26T02:05:33.890 INFO:teuthology.orchestra.run.smithi019.stdout: Qpathinfo 912 1.550 32.093 2024-03-26T02:05:33.890 INFO:teuthology.orchestra.run.smithi019.stdout: Qfileinfo 423 14.108 58.428 2024-03-26T02:05:33.890 INFO:teuthology.orchestra.run.smithi019.stdout: Qfsinfo 165 0.510 3.028 2024-03-26T02:05:33.890 INFO:teuthology.orchestra.run.smithi019.stdout: Sfileinfo 170 12.015 51.029 2024-03-26T02:05:33.890 INFO:teuthology.orchestra.run.smithi019.stdout: Find 336 0.868 1.955 2024-03-26T02:05:33.890 INFO:teuthology.orchestra.run.smithi019.stdout: WriteX 1724 0.024 0.065 2024-03-26T02:05:33.890 INFO:teuthology.orchestra.run.smithi019.stdout: ReadX 1432 0.014 0.925 2024-03-26T02:05:33.891 INFO:teuthology.orchestra.run.smithi019.stdout: Flush 138 12.354 53.285 2024-03-26T02:05:33.891 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-26T02:05:33.891 INFO:teuthology.orchestra.run.smithi019.stdout:Throughput 17.9201 MB/sec 5 clients 5 procs max_latency=65.555 ms 2024-03-26T02:05:33.892 DEBUG:teuthology.parallel:result is None 2024-03-26T02:05:34.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:33 smithi069 ceph-mon[53087]: pgmap v300: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 328 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 11 MiB/s wr, 464 op/s; 184/1311 objects degraded (14.035%) 2024-03-26T02:05:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:35 smithi069 ceph-mon[53087]: pgmap v301: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 320 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 13 MiB/s wr, 501 op/s; 199/1434 objects degraded (13.877%) 2024-03-26T02:05:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:35 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:35 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:36.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:35 smithi019 ceph-mon[66359]: pgmap v301: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 320 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 13 MiB/s wr, 501 op/s; 199/1434 objects degraded (13.877%) 2024-03-26T02:05:36.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:35 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:36.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:35 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:37.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:36 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:37.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:36 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:37.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:36 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:37.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:36 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:38.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:37 smithi069 ceph-mon[53087]: pgmap v302: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 321 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 11 MiB/s wr, 469 op/s; 199/1452 objects degraded (13.705%) 2024-03-26T02:05:38.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:37 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 199/1452 objects degraded (13.705%), 35 pgs degraded (PG_DEGRADED) 2024-03-26T02:05:38.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:37 smithi019 ceph-mon[66359]: pgmap v302: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 321 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 11 MiB/s wr, 469 op/s; 199/1452 objects degraded (13.705%) 2024-03-26T02:05:38.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:37 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 199/1452 objects degraded (13.705%), 35 pgs degraded (PG_DEGRADED) 2024-03-26T02:05:38.999 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:38 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:05:38.746+0000 7f024debe7c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-03-26T02:05:40.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:40 smithi019 ceph-mon[66359]: pgmap v303: 97 pgs: 11 active+undersized, 31 active+undersized+degraded, 55 active+clean; 251 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 11 MiB/s wr, 416 op/s; 92/795 objects degraded (11.572%) 2024-03-26T02:05:40.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:40.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:40.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:05:40.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:05:40.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:40.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:40.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:05:40.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:05:40.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:05:40.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:05:40.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:05:40.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:40.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-26T02:05:40.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-26T02:05:40.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-26T02:05:40.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-26T02:05:40.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-26T02:05:40.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-26T02:05:40.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-26T02:05:40.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-26T02:05:40.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-26T02:05:40.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-26T02:05:40.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-26T02:05:40.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-26T02:05:40.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-26T02:05:40.268 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-26T02:05:40.268 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-26T02:05:40.268 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-26T02:05:40.268 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:40 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-26T02:05:40.359 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:40 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:05:40.228+0000 7f024debe7c0 -1 osd.7 157 log_to_monitors true 2024-03-26T02:05:40.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:40 smithi069 ceph-mon[53087]: pgmap v303: 97 pgs: 11 active+undersized, 31 active+undersized+degraded, 55 active+clean; 251 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 11 MiB/s wr, 416 op/s; 92/795 objects degraded (11.572%) 2024-03-26T02:05:40.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:40.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:40.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:05:40.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:05:40.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:40.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:40.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:05:40.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:05:40.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:05:40.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:05:40.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:05:40.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:40.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-26T02:05:40.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-26T02:05:40.361 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-26T02:05:40.361 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-26T02:05:40.361 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-26T02:05:40.361 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-26T02:05:40.361 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-26T02:05:40.361 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-26T02:05:40.361 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-26T02:05:40.361 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-26T02:05:40.361 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-26T02:05:40.361 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-26T02:05:40.361 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-26T02:05:40.361 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-26T02:05:40.361 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-26T02:05:40.361 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-26T02:05:40.361 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:40 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-26T02:05:41.016 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:05:40 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:05:40.728+0000 7f023f316700 -1 osd.7 157 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:05:41.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:41 smithi019 ceph-mon[66359]: Upgrade: Setting container_image for all osd 2024-03-26T02:05:41.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:41 smithi019 ceph-mon[66359]: Upgrade: Setting require_osd_release to 19 squid 2024-03-26T02:05:41.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:41 smithi019 ceph-mon[66359]: from='osd.7 [v2:172.21.15.69:6824/1978918416,v1:172.21.15.69:6825/1978918416]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T02:05:41.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:41 smithi019 ceph-mon[66359]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T02:05:41.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:41 smithi019 ceph-mon[66359]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-26T02:05:41.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:41 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-26T02:05:41.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:41 smithi019 ceph-mon[66359]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T02:05:41.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:41 smithi019 ceph-mon[66359]: osdmap e160: 8 total, 7 up, 8 in 2024-03-26T02:05:41.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:41 smithi019 ceph-mon[66359]: from='osd.7 [v2:172.21.15.69:6824/1978918416,v1:172.21.15.69:6825/1978918416]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-03-26T02:05:41.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:41 smithi019 ceph-mon[66359]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-03-26T02:05:41.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:41 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi019.lmryom"]}]: dispatch 2024-03-26T02:05:41.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:41 smithi069 ceph-mon[53087]: Upgrade: Setting container_image for all osd 2024-03-26T02:05:41.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:41 smithi069 ceph-mon[53087]: Upgrade: Setting require_osd_release to 19 squid 2024-03-26T02:05:41.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:41 smithi069 ceph-mon[53087]: from='osd.7 [v2:172.21.15.69:6824/1978918416,v1:172.21.15.69:6825/1978918416]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T02:05:41.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:41 smithi069 ceph-mon[53087]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T02:05:41.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:41 smithi069 ceph-mon[53087]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-26T02:05:41.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:41 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-26T02:05:41.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:41 smithi069 ceph-mon[53087]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T02:05:41.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:41 smithi069 ceph-mon[53087]: osdmap e160: 8 total, 7 up, 8 in 2024-03-26T02:05:41.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:41 smithi069 ceph-mon[53087]: from='osd.7 [v2:172.21.15.69:6824/1978918416,v1:172.21.15.69:6825/1978918416]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-03-26T02:05:41.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:41 smithi069 ceph-mon[53087]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-03-26T02:05:41.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:41 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi019.lmryom"]}]: dispatch 2024-03-26T02:05:42.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:42 smithi019 ceph-mon[66359]: Upgrade: It appears safe to stop mds.foofs.smithi019.lmryom 2024-03-26T02:05:42.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:42 smithi019 ceph-mon[66359]: pgmap v305: 97 pgs: 11 active+undersized, 31 active+undersized+degraded, 55 active+clean; 219 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 10 MiB/s wr, 436 op/s; 71/489 objects degraded (14.519%) 2024-03-26T02:05:42.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:42 smithi019 ceph-mon[66359]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:05:42.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:42 smithi019 ceph-mon[66359]: osd.7 [v2:172.21.15.69:6824/1978918416,v1:172.21.15.69:6825/1978918416] boot 2024-03-26T02:05:42.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:42 smithi019 ceph-mon[66359]: osdmap e161: 8 total, 8 up, 8 in 2024-03-26T02:05:42.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:42 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:05:42.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:42 smithi069 ceph-mon[53087]: Upgrade: It appears safe to stop mds.foofs.smithi019.lmryom 2024-03-26T02:05:42.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:42 smithi069 ceph-mon[53087]: pgmap v305: 97 pgs: 11 active+undersized, 31 active+undersized+degraded, 55 active+clean; 219 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 10 MiB/s wr, 436 op/s; 71/489 objects degraded (14.519%) 2024-03-26T02:05:42.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:42 smithi069 ceph-mon[53087]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:05:42.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:42 smithi069 ceph-mon[53087]: osd.7 [v2:172.21.15.69:6824/1978918416,v1:172.21.15.69:6825/1978918416] boot 2024-03-26T02:05:42.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:42 smithi069 ceph-mon[53087]: osdmap e161: 8 total, 8 up, 8 in 2024-03-26T02:05:42.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:42 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:05:43.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:43 smithi019 ceph-mon[66359]: Upgrade: Updating mds.foofs.smithi019.lmryom 2024-03-26T02:05:43.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:43 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:43.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:43 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi019.lmryom", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T02:05:43.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:43 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:05:43.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:43 smithi019 ceph-mon[66359]: Deploying daemon mds.foofs.smithi019.lmryom on smithi019 2024-03-26T02:05:43.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:43 smithi019 ceph-mon[66359]: osdmap e162: 8 total, 8 up, 8 in 2024-03-26T02:05:43.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:43 smithi019 ceph-mon[66359]: pgmap v308: 97 pgs: 8 peering, 7 active+undersized, 27 active+undersized+degraded, 55 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 767 KiB/s wr, 137 op/s; 68/420 objects degraded (16.190%) 2024-03-26T02:05:43.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:43 smithi069 ceph-mon[53087]: Upgrade: Updating mds.foofs.smithi019.lmryom 2024-03-26T02:05:43.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:43 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:43.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:43 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi019.lmryom", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T02:05:43.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:43 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:05:43.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:43 smithi069 ceph-mon[53087]: Deploying daemon mds.foofs.smithi019.lmryom on smithi019 2024-03-26T02:05:43.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:43 smithi069 ceph-mon[53087]: osdmap e162: 8 total, 8 up, 8 in 2024-03-26T02:05:43.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:43 smithi069 ceph-mon[53087]: pgmap v308: 97 pgs: 8 peering, 7 active+undersized, 27 active+undersized+degraded, 55 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 767 KiB/s wr, 137 op/s; 68/420 objects degraded (16.190%) 2024-03-26T02:05:44.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:44 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 68/420 objects degraded (16.190%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T02:05:44.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:44 smithi019 ceph-mon[66359]: osdmap e163: 8 total, 8 up, 8 in 2024-03-26T02:05:44.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:44 smithi019 ceph-mon[66359]: osdmap e164: 8 total, 8 up, 8 in 2024-03-26T02:05:44.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:44 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 68/420 objects degraded (16.190%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T02:05:44.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:44 smithi069 ceph-mon[53087]: osdmap e163: 8 total, 8 up, 8 in 2024-03-26T02:05:44.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:44 smithi069 ceph-mon[53087]: osdmap e164: 8 total, 8 up, 8 in 2024-03-26T02:05:44.918 INFO:teuthology.orchestra.run.smithi019.stdout:true 2024-03-26T02:05:45.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:44 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi019[66355]: 2024-03-26T02:05:44.821+0000 7fa5e31f3700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-26T02:05:45.359 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:05:44 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:05:44.896+0000 7f9352ae0700 -1 bdev(0x563c642a0000 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 5 2024-03-26T02:05:45.621 INFO:teuthology.orchestra.run.smithi019.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:05:45.621 INFO:teuthology.orchestra.run.smithi019.stdout:alertmanager.smithi019 smithi019 *:9093,9094 running (9m) 2m ago 18m 14.5M - 0.23.0 ba2b418f427c a6630841c2b2 2024-03-26T02:05:45.621 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi019 smithi019 running (5m) 2m ago 18m 6660k - 19.0.0-1775-g5e70124b 5f21c9264136 c5f91a47ddf2 2024-03-26T02:05:45.621 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi069 smithi069 running (5m) 10s ago 18m 6669k - 19.0.0-1775-g5e70124b 5f21c9264136 05128803b49e 2024-03-26T02:05:45.621 INFO:teuthology.orchestra.run.smithi019.stdout:grafana.smithi019 smithi019 *:3000 running (9m) 2m ago 18m 46.9M - 8.3.5 dad864ee21e9 a97dea3b5fec 2024-03-26T02:05:45.621 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi019.lmryom smithi019 running (12m) 2m ago 12m 38.4M - 17.2.0 e1d6a67b021e 91a7c2942715 2024-03-26T02:05:45.621 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi069.spgwgo smithi069 running (12m) 10s ago 12m 15.5M - 17.2.0 e1d6a67b021e 94d96338dbe1 2024-03-26T02:05:45.621 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi019.ixuvdm smithi019 *:8443,9283,8765 running (8m) 2m ago 20m 508M - 19.0.0-1775-g5e70124b 5f21c9264136 4460b22c7a56 2024-03-26T02:05:45.621 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi069.wpbewu smithi069 *:8443,9283,8765 running (7m) 10s ago 18m 454M - 19.0.0-1775-g5e70124b 5f21c9264136 380fd33842a8 2024-03-26T02:05:45.621 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi019 smithi019 running (7m) 2m ago 20m 52.0M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 c18934705774 2024-03-26T02:05:45.622 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi069 smithi069 running (6m) 10s ago 18m 44.5M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 a856bd263104 2024-03-26T02:05:45.622 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.0.0.smithi019.qbizxo smithi019 *:2049 running (12m) 2m ago 12m 283M - 4.0 e1d6a67b021e cd6628a9075d 2024-03-26T02:05:45.622 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.1.0.smithi069.uawdhc smithi069 *:2049 running (12m) 10s ago 12m 85.8M - 4.0 e1d6a67b021e 6ee3695e7a10 2024-03-26T02:05:45.622 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi019 smithi019 *:9100 running (9m) 2m ago 18m 10.2M - 1.3.1 1dbe0e931976 0d1d184bec63 2024-03-26T02:05:45.622 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi069 smithi069 *:9100 running (8m) 10s ago 17m 21.5M - 1.3.1 1dbe0e931976 c49a006f94e7 2024-03-26T02:05:45.622 INFO:teuthology.orchestra.run.smithi019.stdout:osd.0 smithi019 running (5m) 2m ago 17m 340M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 3f931cfe320a 2024-03-26T02:05:45.622 INFO:teuthology.orchestra.run.smithi019.stdout:osd.1 smithi019 running (4m) 2m ago 16m 289M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 758caf4ed841 2024-03-26T02:05:45.622 INFO:teuthology.orchestra.run.smithi019.stdout:osd.2 smithi019 running (3m) 2m ago 16m 228M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 6a9b701517ca 2024-03-26T02:05:45.622 INFO:teuthology.orchestra.run.smithi019.stdout:osd.3 smithi019 running (2m) 2m ago 15m 12.6M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 0401940d882b 2024-03-26T02:05:45.622 INFO:teuthology.orchestra.run.smithi019.stdout:osd.4 smithi069 running (2m) 10s ago 15m 341M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 6eb3b2ce9978 2024-03-26T02:05:45.622 INFO:teuthology.orchestra.run.smithi019.stdout:osd.5 smithi069 running (92s) 10s ago 15m 322M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 fb31f0e68fb7 2024-03-26T02:05:45.622 INFO:teuthology.orchestra.run.smithi019.stdout:osd.6 smithi069 running (55s) 10s ago 14m 290M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 924e2a35053c 2024-03-26T02:05:45.622 INFO:teuthology.orchestra.run.smithi019.stdout:osd.7 smithi069 running (16s) 10s ago 14m 12.4M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 d8052b0a18d6 2024-03-26T02:05:45.623 INFO:teuthology.orchestra.run.smithi019.stdout:prometheus.smithi019 smithi019 *:9095 running (7m) 2m ago 17m 67.6M - 2.33.4 514e6a882f6e 29f89db72a07 2024-03-26T02:05:45.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:45 smithi019 ceph-mon[66359]: pgmap v311: 97 pgs: 1 active+recovery_wait+degraded, 1 remapped+peering, 8 peering, 2 active+undersized, 16 active+undersized+degraded, 69 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 73 KiB/s wr, 8 op/s; 69/420 objects degraded (16.429%); 0 B/s, 12 objects/s recovering 2024-03-26T02:05:45.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:45 smithi019 ceph-mon[66359]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-26T02:05:45.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:45 smithi019 ceph-mon[66359]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-26T02:05:45.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:45 smithi019 ceph-mon[66359]: osdmap e165: 8 total, 8 up, 8 in 2024-03-26T02:05:45.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:45 smithi019 ceph-mon[66359]: Standby daemon mds.foofs.smithi069.spgwgo assigned to filesystem foofs as rank 0 2024-03-26T02:05:45.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:45 smithi019 ceph-mon[66359]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-26T02:05:45.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:45 smithi019 ceph-mon[66359]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-26T02:05:45.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:45 smithi019 ceph-mon[66359]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-03-26T02:05:45.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:45 smithi019 ceph-mon[66359]: fsmap foofs:1/1 {0=foofs.smithi069.spgwgo=up:replay} 2024-03-26T02:05:45.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:45 smithi019 ceph-mon[66359]: from='client.34454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:05:45.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:45 smithi069 ceph-mon[53087]: pgmap v311: 97 pgs: 1 active+recovery_wait+degraded, 1 remapped+peering, 8 peering, 2 active+undersized, 16 active+undersized+degraded, 69 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 73 KiB/s wr, 8 op/s; 69/420 objects degraded (16.429%); 0 B/s, 12 objects/s recovering 2024-03-26T02:05:45.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:45 smithi069 ceph-mon[53087]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-26T02:05:45.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:45 smithi069 ceph-mon[53087]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-26T02:05:45.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:45 smithi069 ceph-mon[53087]: osdmap e165: 8 total, 8 up, 8 in 2024-03-26T02:05:45.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:45 smithi069 ceph-mon[53087]: Standby daemon mds.foofs.smithi069.spgwgo assigned to filesystem foofs as rank 0 2024-03-26T02:05:45.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:45 smithi069 ceph-mon[53087]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-26T02:05:45.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:45 smithi069 ceph-mon[53087]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-26T02:05:45.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:45 smithi069 ceph-mon[53087]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-03-26T02:05:45.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:45 smithi069 ceph-mon[53087]: fsmap foofs:1/1 {0=foofs.smithi069.spgwgo=up:replay} 2024-03-26T02:05:45.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:45 smithi069 ceph-mon[53087]: from='client.34454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:05:46.045 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:05:46.045 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": { 2024-03-26T02:05:46.045 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:05:46.045 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:05:46.046 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": { 2024-03-26T02:05:46.046 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:05:46.046 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:05:46.046 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": { 2024-03-26T02:05:46.046 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 8 2024-03-26T02:05:46.046 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:05:46.046 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": { 2024-03-26T02:05:46.046 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1 2024-03-26T02:05:46.046 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:05:46.046 INFO:teuthology.orchestra.run.smithi019.stdout: "overall": { 2024-03-26T02:05:46.046 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-26T02:05:46.046 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 12 2024-03-26T02:05:46.046 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-26T02:05:46.047 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:05:46.385 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:05:46.385 INFO:teuthology.orchestra.run.smithi019.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2", 2024-03-26T02:05:46.385 INFO:teuthology.orchestra.run.smithi019.stdout: "in_progress": true, 2024-03-26T02:05:46.385 INFO:teuthology.orchestra.run.smithi019.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T02:05:46.385 INFO:teuthology.orchestra.run.smithi019.stdout: "services_complete": [ 2024-03-26T02:05:46.385 INFO:teuthology.orchestra.run.smithi019.stdout: "osd", 2024-03-26T02:05:46.385 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr", 2024-03-26T02:05:46.386 INFO:teuthology.orchestra.run.smithi019.stdout: "crash", 2024-03-26T02:05:46.386 INFO:teuthology.orchestra.run.smithi019.stdout: "mon" 2024-03-26T02:05:46.386 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-26T02:05:46.386 INFO:teuthology.orchestra.run.smithi019.stdout: "progress": "14/23 daemons upgraded", 2024-03-26T02:05:46.386 INFO:teuthology.orchestra.run.smithi019.stdout: "message": "Currently upgrading mds daemons", 2024-03-26T02:05:46.386 INFO:teuthology.orchestra.run.smithi019.stdout: "is_paused": false 2024-03-26T02:05:46.386 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:05:46.804 INFO:teuthology.orchestra.run.smithi019.stdout:HEALTH_WARN 1 filesystem is degraded; insufficient standby MDS daemons available; Degraded data redundancy: 69/420 objects degraded (16.429%), 17 pgs degraded 2024-03-26T02:05:46.804 INFO:teuthology.orchestra.run.smithi019.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-26T02:05:46.804 INFO:teuthology.orchestra.run.smithi019.stdout: fs foofs is degraded 2024-03-26T02:05:46.804 INFO:teuthology.orchestra.run.smithi019.stdout:[WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available 2024-03-26T02:05:46.804 INFO:teuthology.orchestra.run.smithi019.stdout: have 0; want 1 more 2024-03-26T02:05:46.804 INFO:teuthology.orchestra.run.smithi019.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 69/420 objects degraded (16.429%), 17 pgs degraded 2024-03-26T02:05:46.805 INFO:teuthology.orchestra.run.smithi019.stdout: pg 1.0 is active+undersized+degraded, acting [0,6] 2024-03-26T02:05:46.805 INFO:teuthology.orchestra.run.smithi019.stdout: pg 2.0 is active+undersized+degraded, acting [1,0] 2024-03-26T02:05:46.805 INFO:teuthology.orchestra.run.smithi019.stdout: pg 2.5 is active+undersized+degraded, acting [0,4] 2024-03-26T02:05:46.805 INFO:teuthology.orchestra.run.smithi019.stdout: pg 2.7 is active+undersized+degraded, acting [6,2] 2024-03-26T02:05:46.805 INFO:teuthology.orchestra.run.smithi019.stdout: pg 2.8 is active+undersized+degraded, acting [0,1] 2024-03-26T02:05:46.805 INFO:teuthology.orchestra.run.smithi019.stdout: pg 2.b is active+undersized+degraded, acting [4,5] 2024-03-26T02:05:46.805 INFO:teuthology.orchestra.run.smithi019.stdout: pg 2.e is active+undersized+degraded, acting [2,3] 2024-03-26T02:05:46.805 INFO:teuthology.orchestra.run.smithi019.stdout: pg 2.1a is active+undersized+degraded, acting [6,4] 2024-03-26T02:05:46.805 INFO:teuthology.orchestra.run.smithi019.stdout: pg 2.1c is active+undersized+degraded, acting [5,2] 2024-03-26T02:05:46.805 INFO:teuthology.orchestra.run.smithi019.stdout: pg 2.1d is active+undersized+degraded, acting [6,0] 2024-03-26T02:05:46.805 INFO:teuthology.orchestra.run.smithi019.stdout: pg 3.7 is active+recovery_wait+degraded, acting [3,7,0] 2024-03-26T02:05:46.805 INFO:teuthology.orchestra.run.smithi019.stdout: pg 3.d is active+undersized+degraded, acting [5,6] 2024-03-26T02:05:46.805 INFO:teuthology.orchestra.run.smithi019.stdout: pg 3.f is active+undersized+degraded, acting [4,0] 2024-03-26T02:05:46.805 INFO:teuthology.orchestra.run.smithi019.stdout: pg 3.11 is active+undersized+degraded, acting [4,6] 2024-03-26T02:05:46.806 INFO:teuthology.orchestra.run.smithi019.stdout: pg 3.15 is active+undersized+degraded, acting [3,4] 2024-03-26T02:05:46.806 INFO:teuthology.orchestra.run.smithi019.stdout: pg 4.5 is active+undersized+degraded, acting [6,4] 2024-03-26T02:05:46.806 INFO:teuthology.orchestra.run.smithi019.stdout: pg 4.a is active+undersized+degraded, acting [6,2] 2024-03-26T02:05:47.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:46 smithi069 ceph-mon[53087]: from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:05:47.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:46 smithi069 ceph-mon[53087]: from='client.34462 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:05:47.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:46 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:47.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:46 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:05:47.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:46 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/2026022523' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:05:47.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:46 smithi069 ceph-mon[53087]: mds.? [v2:172.21.15.69:6832/641242727,v1:172.21.15.69:6833/641242727] up:reconnect 2024-03-26T02:05:47.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:46 smithi069 ceph-mon[53087]: fsmap foofs:1/1 {0=foofs.smithi069.spgwgo=up:reconnect} 2024-03-26T02:05:47.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:46 smithi019 ceph-mon[66359]: from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:05:47.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:46 smithi019 ceph-mon[66359]: from='client.34462 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:05:47.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:46 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:47.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:46 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:05:47.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:46 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/2026022523' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:05:47.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:46 smithi019 ceph-mon[66359]: mds.? [v2:172.21.15.69:6832/641242727,v1:172.21.15.69:6833/641242727] up:reconnect 2024-03-26T02:05:47.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:46 smithi019 ceph-mon[66359]: fsmap foofs:1/1 {0=foofs.smithi069.spgwgo=up:reconnect} 2024-03-26T02:05:48.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:47 smithi069 ceph-mon[53087]: from='client.34470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:05:48.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:47 smithi069 ceph-mon[53087]: reconnect by client.24353 172.21.15.69:0/1561493568 after 0.000999979 2024-03-26T02:05:48.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:47 smithi069 ceph-mon[53087]: reconnect by client.14536 172.21.15.19:0/1467571761 after 0.000999979 2024-03-26T02:05:48.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:47 smithi069 ceph-mon[53087]: pgmap v313: 97 pgs: 1 active+recovery_wait+degraded, 1 remapped+peering, 8 peering, 87 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 25 MiB/s rd, 8 op/s; 26/420 objects degraded (6.190%); 1020 KiB/s, 53 objects/s recovering 2024-03-26T02:05:48.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:47 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/846275934' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:05:48.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:47 smithi069 ceph-mon[53087]: mds.? [v2:172.21.15.69:6832/641242727,v1:172.21.15.69:6833/641242727] up:rejoin 2024-03-26T02:05:48.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:47 smithi069 ceph-mon[53087]: fsmap foofs:1/1 {0=foofs.smithi069.spgwgo=up:rejoin} 2024-03-26T02:05:48.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:47 smithi069 ceph-mon[53087]: daemon mds.foofs.smithi069.spgwgo is now active in filesystem foofs as rank 0 2024-03-26T02:05:48.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:47 smithi019 ceph-mon[66359]: from='client.34470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:05:48.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:47 smithi019 ceph-mon[66359]: reconnect by client.24353 172.21.15.69:0/1561493568 after 0.000999979 2024-03-26T02:05:48.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:47 smithi019 ceph-mon[66359]: reconnect by client.14536 172.21.15.19:0/1467571761 after 0.000999979 2024-03-26T02:05:48.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:47 smithi019 ceph-mon[66359]: pgmap v313: 97 pgs: 1 active+recovery_wait+degraded, 1 remapped+peering, 8 peering, 87 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 25 MiB/s rd, 8 op/s; 26/420 objects degraded (6.190%); 1020 KiB/s, 53 objects/s recovering 2024-03-26T02:05:48.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:47 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/846275934' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:05:48.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:47 smithi019 ceph-mon[66359]: mds.? [v2:172.21.15.69:6832/641242727,v1:172.21.15.69:6833/641242727] up:rejoin 2024-03-26T02:05:48.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:47 smithi019 ceph-mon[66359]: fsmap foofs:1/1 {0=foofs.smithi069.spgwgo=up:rejoin} 2024-03-26T02:05:48.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:47 smithi019 ceph-mon[66359]: daemon mds.foofs.smithi069.spgwgo is now active in filesystem foofs as rank 0 2024-03-26T02:05:49.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:48 smithi069 ceph-mon[53087]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-26T02:05:49.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:48 smithi069 ceph-mon[53087]: mds.? [v2:172.21.15.69:6832/641242727,v1:172.21.15.69:6833/641242727] up:active 2024-03-26T02:05:49.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:48 smithi069 ceph-mon[53087]: fsmap foofs:1 {0=foofs.smithi069.spgwgo=up:active} 2024-03-26T02:05:49.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:48 smithi019 ceph-mon[66359]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-26T02:05:49.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:48 smithi019 ceph-mon[66359]: mds.? [v2:172.21.15.69:6832/641242727,v1:172.21.15.69:6833/641242727] up:active 2024-03-26T02:05:49.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:48 smithi019 ceph-mon[66359]: fsmap foofs:1 {0=foofs.smithi069.spgwgo=up:active} 2024-03-26T02:05:50.042 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:49 smithi019 ceph-mon[66359]: pgmap v314: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 27 MiB/s rd, 5.8 KiB/s wr, 11 op/s; 1.3 MiB/s, 60 objects/s recovering 2024-03-26T02:05:50.042 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:49 smithi019 ceph-mon[66359]: Health check update: Degraded data redundancy: 26/420 objects degraded (6.190%), 1 pg degraded (PG_DEGRADED) 2024-03-26T02:05:50.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:49 smithi069 ceph-mon[53087]: pgmap v314: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 27 MiB/s rd, 5.8 KiB/s wr, 11 op/s; 1.3 MiB/s, 60 objects/s recovering 2024-03-26T02:05:50.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:49 smithi069 ceph-mon[53087]: Health check update: Degraded data redundancy: 26/420 objects degraded (6.190%), 1 pg degraded (PG_DEGRADED) 2024-03-26T02:05:50.939 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:50 smithi019 ceph-mon[66359]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 26/420 objects degraded (6.190%), 1 pg degraded) 2024-03-26T02:05:51.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:50 smithi069 ceph-mon[53087]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 26/420 objects degraded (6.190%), 1 pg degraded) 2024-03-26T02:05:52.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:51 smithi019 ceph-mon[66359]: pgmap v315: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 30 MiB/s rd, 5.0 KiB/s wr, 11 op/s; 1.1 MiB/s, 48 objects/s recovering 2024-03-26T02:05:52.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:51 smithi069 ceph-mon[53087]: pgmap v315: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 30 MiB/s rd, 5.0 KiB/s wr, 11 op/s; 1.1 MiB/s, 48 objects/s recovering 2024-03-26T02:05:52.980 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:52 smithi019 ceph-mon[66359]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-26T02:05:52.980 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:52 smithi019 ceph-mon[66359]: Cluster is now healthy 2024-03-26T02:05:52.980 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:52 smithi019 ceph-mon[66359]: mds.? [v2:172.21.15.19:6834/2546344341,v1:172.21.15.19:6835/2546344341] up:boot 2024-03-26T02:05:52.981 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:52 smithi019 ceph-mon[66359]: fsmap foofs:1 {0=foofs.smithi069.spgwgo=up:active} 1 up:standby 2024-03-26T02:05:52.981 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:52 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi019.lmryom"}]: dispatch 2024-03-26T02:05:52.981 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:52 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:52.981 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:52 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:52.981 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:52 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:05:53.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:52 smithi069 ceph-mon[53087]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-26T02:05:53.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:52 smithi069 ceph-mon[53087]: Cluster is now healthy 2024-03-26T02:05:53.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:52 smithi069 ceph-mon[53087]: mds.? [v2:172.21.15.19:6834/2546344341,v1:172.21.15.19:6835/2546344341] up:boot 2024-03-26T02:05:53.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:52 smithi069 ceph-mon[53087]: fsmap foofs:1 {0=foofs.smithi069.spgwgo=up:active} 1 up:standby 2024-03-26T02:05:53.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:52 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi019.lmryom"}]: dispatch 2024-03-26T02:05:53.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:52 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:53.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:52 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:53.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:52 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:05:54.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:53 smithi069 ceph-mon[53087]: pgmap v316: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 25 MiB/s rd, 14 KiB/s wr, 18 op/s; 976 KiB/s, 37 objects/s recovering 2024-03-26T02:05:54.172 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:53 smithi019 ceph-mon[66359]: pgmap v316: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 25 MiB/s rd, 14 KiB/s wr, 18 op/s; 976 KiB/s, 37 objects/s recovering 2024-03-26T02:05:55.935 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:55 smithi019 ceph-mon[66359]: pgmap v317: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 12 KiB/s wr, 15 op/s; 819 KiB/s, 31 objects/s recovering 2024-03-26T02:05:56.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:55 smithi069 ceph-mon[53087]: pgmap v317: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 12 KiB/s wr, 15 op/s; 819 KiB/s, 31 objects/s recovering 2024-03-26T02:05:57.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:56 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:57.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:56 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:58.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:58 smithi069 ceph-mon[53087]: pgmap v318: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 10 KiB/s wr, 13 op/s; 685 KiB/s, 26 objects/s recovering 2024-03-26T02:05:58.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:58 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:58.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:58 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:58.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:58 smithi019 ceph-mon[66359]: pgmap v318: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 10 KiB/s wr, 13 op/s; 685 KiB/s, 26 objects/s recovering 2024-03-26T02:05:58.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:58 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:58.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:58 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:59.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:59 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:59.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:59 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:59.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:05:59 smithi019 ceph-mon[66359]: pgmap v319: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.0 MiB/s rd, 10 KiB/s wr, 10 op/s; 341 KiB/s, 12 objects/s recovering 2024-03-26T02:05:59.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:59 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:59.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:59 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:05:59.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:05:59 smithi069 ceph-mon[53087]: pgmap v319: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.0 MiB/s rd, 10 KiB/s wr, 10 op/s; 341 KiB/s, 12 objects/s recovering 2024-03-26T02:06:02.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:02.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:06:02.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:01 smithi069 ceph-mon[53087]: pgmap v320: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.0 MiB/s rd, 7.2 KiB/s wr, 7 op/s 2024-03-26T02:06:02.199 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:02.199 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:06:02.199 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:01 smithi019 ceph-mon[66359]: pgmap v320: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.0 MiB/s rd, 7.2 KiB/s wr, 7 op/s 2024-03-26T02:06:03.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:03 smithi069 ceph-mon[53087]: Detected new or changed devices on smithi019 2024-03-26T02:06:03.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:03 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:03.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:03 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:03.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:03 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:06:03.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:03 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:06:03.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:03 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:03.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:03 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:03.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:03 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:06:03.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:03 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:03.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:03 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:03.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:03 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:03.611 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:03 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:03.611 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:03 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi069.spgwgo"]}]: dispatch 2024-03-26T02:06:03.611 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:03 smithi069 ceph-mon[53087]: Upgrade: It appears safe to stop mds.foofs.smithi069.spgwgo 2024-03-26T02:06:03.611 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:03 smithi069 ceph-mon[53087]: pgmap v321: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 7.3 KiB/s wr, 6 op/s 2024-03-26T02:06:03.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:03 smithi019 ceph-mon[66359]: Detected new or changed devices on smithi019 2024-03-26T02:06:03.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:03 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:03.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:03 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:03.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:03 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:06:03.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:03 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:06:03.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:03 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:03.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:03 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:03.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:03 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:06:03.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:03 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:03.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:03 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:03.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:03 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:03.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:03 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:03.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:03 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi069.spgwgo"]}]: dispatch 2024-03-26T02:06:03.767 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:03 smithi019 ceph-mon[66359]: Upgrade: It appears safe to stop mds.foofs.smithi069.spgwgo 2024-03-26T02:06:03.767 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:03 smithi019 ceph-mon[66359]: pgmap v321: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 7.3 KiB/s wr, 6 op/s 2024-03-26T02:06:05.236 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:05 smithi069 ceph-mon[53087]: Upgrade: Updating mds.foofs.smithi069.spgwgo 2024-03-26T02:06:05.236 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:05.236 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi069.spgwgo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T02:06:05.236 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:06:05.236 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:05 smithi069 ceph-mon[53087]: Deploying daemon mds.foofs.smithi069.spgwgo on smithi069 2024-03-26T02:06:05.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:05 smithi019 ceph-mon[66359]: Upgrade: Updating mds.foofs.smithi069.spgwgo 2024-03-26T02:06:05.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:05.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi069.spgwgo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T02:06:05.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:06:05.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:05 smithi019 ceph-mon[66359]: Deploying daemon mds.foofs.smithi069.spgwgo on smithi069 2024-03-26T02:06:06.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:06 smithi069 ceph-mon[53087]: pgmap v322: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 170 B/s wr, 0 op/s 2024-03-26T02:06:06.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:06 smithi019 ceph-mon[66359]: pgmap v322: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 170 B/s wr, 0 op/s 2024-03-26T02:06:06.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:06 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi019[66355]: 2024-03-26T02:06:06.041+0000 7fa5e31f3700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-26T02:06:06.609 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:06:06 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:06:06.123+0000 7f9352ae0700 -1 bdev(0x563c642a0000 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 3 2024-03-26T02:06:06.610 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:06:06 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:06:06.124+0000 7f93592ed700 -1 bdev(0x563c642a0000 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 8 2024-03-26T02:06:07.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:07 smithi019 ceph-mon[66359]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-26T02:06:07.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:07 smithi019 ceph-mon[66359]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-26T02:06:07.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:07 smithi019 ceph-mon[66359]: osdmap e166: 8 total, 8 up, 8 in 2024-03-26T02:06:07.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:07 smithi019 ceph-mon[66359]: Standby daemon mds.foofs.smithi019.lmryom assigned to filesystem foofs as rank 0 2024-03-26T02:06:07.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:07 smithi019 ceph-mon[66359]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-26T02:06:07.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:07 smithi019 ceph-mon[66359]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-26T02:06:07.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:07 smithi019 ceph-mon[66359]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-03-26T02:06:07.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:07 smithi019 ceph-mon[66359]: fsmap foofs:1/1 {0=foofs.smithi019.lmryom=up:replay} 2024-03-26T02:06:07.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:07 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi069.spgwgo"}]: dispatch 2024-03-26T02:06:07.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:07 smithi069 ceph-mon[53087]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-26T02:06:07.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:07 smithi069 ceph-mon[53087]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-26T02:06:07.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:07 smithi069 ceph-mon[53087]: osdmap e166: 8 total, 8 up, 8 in 2024-03-26T02:06:07.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:07 smithi069 ceph-mon[53087]: Standby daemon mds.foofs.smithi019.lmryom assigned to filesystem foofs as rank 0 2024-03-26T02:06:07.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:07 smithi069 ceph-mon[53087]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-26T02:06:07.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:07 smithi069 ceph-mon[53087]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-26T02:06:07.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:07 smithi069 ceph-mon[53087]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-03-26T02:06:07.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:07 smithi069 ceph-mon[53087]: fsmap foofs:1/1 {0=foofs.smithi019.lmryom=up:replay} 2024-03-26T02:06:07.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:07 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi069.spgwgo"}]: dispatch 2024-03-26T02:06:08.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:08 smithi069 ceph-mon[53087]: pgmap v324: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 204 B/s wr, 0 op/s 2024-03-26T02:06:08.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:08 smithi069 ceph-mon[53087]: reconnect by client.14536 172.21.15.19:0/1467571761 after 0 2024-03-26T02:06:08.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:08 smithi069 ceph-mon[53087]: reconnect by client.24353 172.21.15.69:0/1561493568 after 0.00099999 2024-03-26T02:06:08.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:08 smithi069 ceph-mon[53087]: mds.? [v2:172.21.15.19:6834/2546344341,v1:172.21.15.19:6835/2546344341] up:reconnect 2024-03-26T02:06:08.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:08 smithi069 ceph-mon[53087]: fsmap foofs:1/1 {0=foofs.smithi019.lmryom=up:reconnect} 2024-03-26T02:06:08.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:08 smithi019 ceph-mon[66359]: pgmap v324: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 204 B/s wr, 0 op/s 2024-03-26T02:06:08.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:08 smithi019 ceph-mon[66359]: reconnect by client.14536 172.21.15.19:0/1467571761 after 0 2024-03-26T02:06:08.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:08 smithi019 ceph-mon[66359]: reconnect by client.24353 172.21.15.69:0/1561493568 after 0.00099999 2024-03-26T02:06:08.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:08 smithi019 ceph-mon[66359]: mds.? [v2:172.21.15.19:6834/2546344341,v1:172.21.15.19:6835/2546344341] up:reconnect 2024-03-26T02:06:08.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:08 smithi019 ceph-mon[66359]: fsmap foofs:1/1 {0=foofs.smithi019.lmryom=up:reconnect} 2024-03-26T02:06:09.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:09 smithi069 ceph-mon[53087]: mds.? [v2:172.21.15.19:6834/2546344341,v1:172.21.15.19:6835/2546344341] up:rejoin 2024-03-26T02:06:09.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:09 smithi069 ceph-mon[53087]: fsmap foofs:1/1 {0=foofs.smithi019.lmryom=up:rejoin} 2024-03-26T02:06:09.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:09 smithi069 ceph-mon[53087]: daemon mds.foofs.smithi019.lmryom is now active in filesystem foofs as rank 0 2024-03-26T02:06:09.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:09 smithi019 ceph-mon[66359]: mds.? [v2:172.21.15.19:6834/2546344341,v1:172.21.15.19:6835/2546344341] up:rejoin 2024-03-26T02:06:09.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:09 smithi019 ceph-mon[66359]: fsmap foofs:1/1 {0=foofs.smithi019.lmryom=up:rejoin} 2024-03-26T02:06:09.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:09 smithi019 ceph-mon[66359]: daemon mds.foofs.smithi019.lmryom is now active in filesystem foofs as rank 0 2024-03-26T02:06:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:10 smithi069 ceph-mon[53087]: pgmap v325: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 6.0 MiB/s rd, 204 B/s wr, 2 op/s 2024-03-26T02:06:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:10 smithi069 ceph-mon[53087]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-26T02:06:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:10 smithi069 ceph-mon[53087]: mds.? [v2:172.21.15.19:6834/2546344341,v1:172.21.15.19:6835/2546344341] up:active 2024-03-26T02:06:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:10 smithi069 ceph-mon[53087]: fsmap foofs:1 {0=foofs.smithi019.lmryom=up:active} 2024-03-26T02:06:10.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:10 smithi019 ceph-mon[66359]: pgmap v325: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 6.0 MiB/s rd, 204 B/s wr, 2 op/s 2024-03-26T02:06:10.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:10 smithi019 ceph-mon[66359]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-26T02:06:10.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:10 smithi019 ceph-mon[66359]: mds.? [v2:172.21.15.19:6834/2546344341,v1:172.21.15.19:6835/2546344341] up:active 2024-03-26T02:06:10.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:10 smithi019 ceph-mon[66359]: fsmap foofs:1 {0=foofs.smithi019.lmryom=up:active} 2024-03-26T02:06:12.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:12 smithi069 ceph-mon[53087]: pgmap v326: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 204 B/s wr, 4 op/s 2024-03-26T02:06:12.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:12 smithi019 ceph-mon[66359]: pgmap v326: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 204 B/s wr, 4 op/s 2024-03-26T02:06:14.241 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:14 smithi069 ceph-mon[53087]: pgmap v327: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 8.5 KiB/s wr, 14 op/s 2024-03-26T02:06:14.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:14 smithi019 ceph-mon[66359]: pgmap v327: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 8.5 KiB/s wr, 14 op/s 2024-03-26T02:06:16.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:16 smithi069 ceph-mon[53087]: pgmap v328: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 8.5 KiB/s wr, 14 op/s 2024-03-26T02:06:16.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:16 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:16.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:16 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:16.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:16 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:06:16.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:16 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:16.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:16 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:06:16.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:16 smithi019 ceph-mon[66359]: pgmap v328: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 8.5 KiB/s wr, 14 op/s 2024-03-26T02:06:16.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:16 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:16.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:16 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:16.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:16 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:06:16.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:16 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:16.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:16 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:06:17.165 INFO:teuthology.orchestra.run.smithi019.stdout:true 2024-03-26T02:06:17.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:17 smithi069 ceph-mon[53087]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-26T02:06:17.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:17 smithi069 ceph-mon[53087]: Cluster is now healthy 2024-03-26T02:06:17.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:17 smithi069 ceph-mon[53087]: mds.? [v2:172.21.15.69:6832/3949293973,v1:172.21.15.69:6833/3949293973] up:boot 2024-03-26T02:06:17.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:17 smithi069 ceph-mon[53087]: fsmap foofs:1 {0=foofs.smithi019.lmryom=up:active} 1 up:standby 2024-03-26T02:06:17.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:17 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi069.spgwgo"}]: dispatch 2024-03-26T02:06:17.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:17 smithi019 ceph-mon[66359]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-26T02:06:17.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:17 smithi019 ceph-mon[66359]: Cluster is now healthy 2024-03-26T02:06:17.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:17 smithi019 ceph-mon[66359]: mds.? [v2:172.21.15.69:6832/3949293973,v1:172.21.15.69:6833/3949293973] up:boot 2024-03-26T02:06:17.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:17 smithi019 ceph-mon[66359]: fsmap foofs:1 {0=foofs.smithi019.lmryom=up:active} 1 up:standby 2024-03-26T02:06:17.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:17 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi069.spgwgo"}]: dispatch 2024-03-26T02:06:17.910 INFO:teuthology.orchestra.run.smithi019.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:06:17.911 INFO:teuthology.orchestra.run.smithi019.stdout:alertmanager.smithi019 smithi019 *:9093,9094 running (10m) 20s ago 19m 24.9M - 0.23.0 ba2b418f427c a6630841c2b2 2024-03-26T02:06:17.911 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi019 smithi019 running (6m) 20s ago 19m 6660k - 19.0.0-1775-g5e70124b 5f21c9264136 c5f91a47ddf2 2024-03-26T02:06:17.911 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi069 smithi069 running (6m) 43s ago 18m 6669k - 19.0.0-1775-g5e70124b 5f21c9264136 05128803b49e 2024-03-26T02:06:17.911 INFO:teuthology.orchestra.run.smithi019.stdout:grafana.smithi019 smithi019 *:3000 running (9m) 20s ago 19m 48.2M - 8.3.5 dad864ee21e9 a97dea3b5fec 2024-03-26T02:06:17.911 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi019.lmryom smithi019 running (26s) 20s ago 13m 14.5M - 19.0.0-1775-g5e70124b 5f21c9264136 499227af5537 2024-03-26T02:06:17.911 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi069.spgwgo smithi069 starting - - - - 2024-03-26T02:06:17.911 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi019.ixuvdm smithi019 *:8443,9283,8765 running (9m) 20s ago 20m 513M - 19.0.0-1775-g5e70124b 5f21c9264136 4460b22c7a56 2024-03-26T02:06:17.911 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi069.wpbewu smithi069 *:8443,9283,8765 running (8m) 43s ago 18m 454M - 19.0.0-1775-g5e70124b 5f21c9264136 380fd33842a8 2024-03-26T02:06:17.911 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi019 smithi019 running (7m) 20s ago 20m 61.4M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 c18934705774 2024-03-26T02:06:17.911 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi069 smithi069 running (7m) 43s ago 18m 44.5M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 a856bd263104 2024-03-26T02:06:17.911 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.0.0.smithi019.qbizxo smithi019 *:2049 running (13m) 20s ago 13m 268M - 4.0 e1d6a67b021e cd6628a9075d 2024-03-26T02:06:17.912 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.1.0.smithi069.uawdhc smithi069 *:2049 running (12m) 43s ago 12m 85.8M - 4.0 e1d6a67b021e 6ee3695e7a10 2024-03-26T02:06:17.912 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi019 smithi019 *:9100 running (9m) 20s ago 19m 10.4M - 1.3.1 1dbe0e931976 0d1d184bec63 2024-03-26T02:06:17.912 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi069 smithi069 *:9100 running (9m) 43s ago 18m 21.5M - 1.3.1 1dbe0e931976 c49a006f94e7 2024-03-26T02:06:17.912 INFO:teuthology.orchestra.run.smithi019.stdout:osd.0 smithi019 running (5m) 20s ago 17m 408M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 3f931cfe320a 2024-03-26T02:06:17.912 INFO:teuthology.orchestra.run.smithi019.stdout:osd.1 smithi019 running (5m) 20s ago 17m 371M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 758caf4ed841 2024-03-26T02:06:17.912 INFO:teuthology.orchestra.run.smithi019.stdout:osd.2 smithi019 running (4m) 20s ago 16m 380M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 6a9b701517ca 2024-03-26T02:06:17.912 INFO:teuthology.orchestra.run.smithi019.stdout:osd.3 smithi019 running (3m) 20s ago 16m 419M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 0401940d882b 2024-03-26T02:06:17.912 INFO:teuthology.orchestra.run.smithi019.stdout:osd.4 smithi069 running (2m) 43s ago 16m 341M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 6eb3b2ce9978 2024-03-26T02:06:17.912 INFO:teuthology.orchestra.run.smithi019.stdout:osd.5 smithi069 running (2m) 43s ago 15m 322M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 fb31f0e68fb7 2024-03-26T02:06:17.912 INFO:teuthology.orchestra.run.smithi019.stdout:osd.6 smithi069 running (87s) 43s ago 15m 290M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 924e2a35053c 2024-03-26T02:06:17.912 INFO:teuthology.orchestra.run.smithi019.stdout:osd.7 smithi069 running (49s) 43s ago 14m 12.4M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 d8052b0a18d6 2024-03-26T02:06:17.912 INFO:teuthology.orchestra.run.smithi019.stdout:prometheus.smithi019 smithi019 *:9095 running (8m) 20s ago 18m 69.6M - 2.33.4 514e6a882f6e 29f89db72a07 2024-03-26T02:06:18.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:18 smithi019 ceph-mon[66359]: pgmap v329: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 7.9 KiB/s wr, 13 op/s 2024-03-26T02:06:18.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:18 smithi069 ceph-mon[53087]: pgmap v329: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 7.9 KiB/s wr, 13 op/s 2024-03-26T02:06:18.389 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:06:18.389 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": { 2024-03-26T02:06:18.389 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:06:18.389 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:06:18.389 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": { 2024-03-26T02:06:18.389 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:06:18.390 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:06:18.390 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": { 2024-03-26T02:06:18.390 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 8 2024-03-26T02:06:18.390 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:06:18.390 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": { 2024-03-26T02:06:18.390 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:06:18.390 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:06:18.390 INFO:teuthology.orchestra.run.smithi019.stdout: "overall": { 2024-03-26T02:06:18.390 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 14 2024-03-26T02:06:18.390 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-26T02:06:18.390 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:06:18.766 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:06:18.766 INFO:teuthology.orchestra.run.smithi019.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2", 2024-03-26T02:06:18.766 INFO:teuthology.orchestra.run.smithi019.stdout: "in_progress": true, 2024-03-26T02:06:18.766 INFO:teuthology.orchestra.run.smithi019.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T02:06:18.766 INFO:teuthology.orchestra.run.smithi019.stdout: "services_complete": [ 2024-03-26T02:06:18.766 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr", 2024-03-26T02:06:18.766 INFO:teuthology.orchestra.run.smithi019.stdout: "mds", 2024-03-26T02:06:18.767 INFO:teuthology.orchestra.run.smithi019.stdout: "crash", 2024-03-26T02:06:18.767 INFO:teuthology.orchestra.run.smithi019.stdout: "mon", 2024-03-26T02:06:18.767 INFO:teuthology.orchestra.run.smithi019.stdout: "osd" 2024-03-26T02:06:18.767 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-26T02:06:18.767 INFO:teuthology.orchestra.run.smithi019.stdout: "progress": "16/23 daemons upgraded", 2024-03-26T02:06:18.767 INFO:teuthology.orchestra.run.smithi019.stdout: "message": "Currently upgrading mds daemons", 2024-03-26T02:06:18.767 INFO:teuthology.orchestra.run.smithi019.stdout: "is_paused": false 2024-03-26T02:06:18.767 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:06:19.193 INFO:teuthology.orchestra.run.smithi019.stdout:HEALTH_OK 2024-03-26T02:06:19.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:19 smithi069 ceph-mon[53087]: from='client.34484 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:06:19.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:19 smithi069 ceph-mon[53087]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:06:19.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:19 smithi069 ceph-mon[53087]: from='client.34492 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:06:19.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:19 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:19.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:19 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:19.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:19 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/2621531215' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:19.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:19 smithi019 ceph-mon[66359]: from='client.34484 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:06:19.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:19 smithi019 ceph-mon[66359]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:06:19.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:19 smithi019 ceph-mon[66359]: from='client.34492 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:06:19.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:19 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:19.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:19 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:19.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:19 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/2621531215' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:20 smithi069 ceph-mon[53087]: from='client.34500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:06:20.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:20 smithi069 ceph-mon[53087]: pgmap v330: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 7.3 KiB/s wr, 12 op/s 2024-03-26T02:06:20.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:20 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/1092141230' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:06:20.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:20 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:20.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:20 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:20.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:20 smithi019 ceph-mon[66359]: from='client.34500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:06:20.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:20 smithi019 ceph-mon[66359]: pgmap v330: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 7.3 KiB/s wr, 12 op/s 2024-03-26T02:06:20.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:20 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/1092141230' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:06:20.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:20 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:20.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:20 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:22.327 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:22 smithi069 ceph-mon[53087]: pgmap v331: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 7.3 KiB/s wr, 10 op/s 2024-03-26T02:06:22.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:22 smithi019 ceph-mon[66359]: pgmap v331: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 7.3 KiB/s wr, 10 op/s 2024-03-26T02:06:23.911 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: Detected new or changed devices on smithi069 2024-03-26T02:06:23.911 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:23.911 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:23.912 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:06:23.912 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:06:23.912 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:23.912 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:23.912 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: pgmap v332: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.5 MiB/s rd, 7.5 KiB/s wr, 8 op/s 2024-03-26T02:06:23.912 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:06:23.912 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:23.912 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:23.913 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:23.913 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:23.913 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:23.913 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: Upgrade: Setting container_image for all mds 2024-03-26T02:06:23.913 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:23.913 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi019.lmryom"}]: dispatch 2024-03-26T02:06:23.913 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi019.lmryom"}]': finished 2024-03-26T02:06:23.913 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi069.spgwgo"}]: dispatch 2024-03-26T02:06:23.914 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi069.spgwgo"}]': finished 2024-03-26T02:06:23.914 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:23.914 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: Upgrade: Setting container_image for all rgw 2024-03-26T02:06:23.914 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:23.914 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:23.914 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: Upgrade: Setting container_image for all rbd-mirror 2024-03-26T02:06:23.914 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:23.915 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:23.915 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:23.915 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: Upgrade: Setting container_image for all ceph-exporter 2024-03-26T02:06:23.915 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:23.915 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:23.915 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: Upgrade: Setting container_image for all iscsi 2024-03-26T02:06:23.915 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:23 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:24.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: Detected new or changed devices on smithi069 2024-03-26T02:06:24.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:24.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:24.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:06:24.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:06:24.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:24.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:24.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: pgmap v332: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.5 MiB/s rd, 7.5 KiB/s wr, 8 op/s 2024-03-26T02:06:24.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:06:24.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:24.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:24.112 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:24.112 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:24.112 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:24.112 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: Upgrade: Setting container_image for all mds 2024-03-26T02:06:24.112 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:24.112 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi019.lmryom"}]: dispatch 2024-03-26T02:06:24.112 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi019.lmryom"}]': finished 2024-03-26T02:06:24.112 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi069.spgwgo"}]: dispatch 2024-03-26T02:06:24.112 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi069.spgwgo"}]': finished 2024-03-26T02:06:24.112 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:24.112 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: Upgrade: Setting container_image for all rgw 2024-03-26T02:06:24.112 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:24.113 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:24.113 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: Upgrade: Setting container_image for all rbd-mirror 2024-03-26T02:06:24.113 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:24.113 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:24.113 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:24.113 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: Upgrade: Setting container_image for all ceph-exporter 2024-03-26T02:06:24.113 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:24.113 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:24.113 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: Upgrade: Setting container_image for all iscsi 2024-03-26T02:06:24.113 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:23 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:26.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:25 smithi019 ceph-mon[66359]: pgmap v333: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 426 B/s wr, 0 op/s 2024-03-26T02:06:26.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:25 smithi019 ceph-mon[66359]: Upgrade: Updating nfs.foo.0.0.smithi019.qbizxo (1/2) 2024-03-26T02:06:26.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:25 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:26.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:25 smithi019 ceph-mon[66359]: Creating key for client.nfs.foo.0.0.smithi019.qbizxo 2024-03-26T02:06:26.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:25 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi019.qbizxo", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-26T02:06:26.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:25 smithi019 ceph-mon[66359]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-03-26T02:06:26.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:25 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-26T02:06:26.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:25 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-26T02:06:26.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:25 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:06:26.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:25 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-26T02:06:26.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:25 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-26T02:06:26.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:25 smithi019 ceph-mon[66359]: Rados config object exists: conf-nfs.foo 2024-03-26T02:06:26.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:25 smithi019 ceph-mon[66359]: Creating key for client.nfs.foo.0.0.smithi019.qbizxo-rgw 2024-03-26T02:06:26.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:25 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi019.qbizxo-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T02:06:26.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:25 smithi019 ceph-mon[66359]: Bind address in nfs.foo.0.0.smithi019.qbizxo's ganesha conf is defaulting to empty 2024-03-26T02:06:26.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:25 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:06:26.268 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:25 smithi019 ceph-mon[66359]: Deploying daemon nfs.foo.0.0.smithi019.qbizxo on smithi019 2024-03-26T02:06:26.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:25 smithi069 ceph-mon[53087]: pgmap v333: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 426 B/s wr, 0 op/s 2024-03-26T02:06:26.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:25 smithi069 ceph-mon[53087]: Upgrade: Updating nfs.foo.0.0.smithi019.qbizxo (1/2) 2024-03-26T02:06:26.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:25 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:26.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:25 smithi069 ceph-mon[53087]: Creating key for client.nfs.foo.0.0.smithi019.qbizxo 2024-03-26T02:06:26.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:25 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi019.qbizxo", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-26T02:06:26.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:25 smithi069 ceph-mon[53087]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-03-26T02:06:26.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:25 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-26T02:06:26.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:25 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-26T02:06:26.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:25 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:06:26.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:25 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-26T02:06:26.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:25 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-26T02:06:26.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:25 smithi069 ceph-mon[53087]: Rados config object exists: conf-nfs.foo 2024-03-26T02:06:26.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:25 smithi069 ceph-mon[53087]: Creating key for client.nfs.foo.0.0.smithi019.qbizxo-rgw 2024-03-26T02:06:26.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:25 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi019.qbizxo-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T02:06:26.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:25 smithi069 ceph-mon[53087]: Bind address in nfs.foo.0.0.smithi019.qbizxo's ganesha conf is defaulting to empty 2024-03-26T02:06:26.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:25 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:06:26.360 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:25 smithi069 ceph-mon[53087]: Deploying daemon nfs.foo.0.0.smithi019.qbizxo on smithi019 2024-03-26T02:06:28.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:27 smithi019 ceph-mon[66359]: pgmap v334: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 426 B/s wr, 0 op/s 2024-03-26T02:06:28.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:27 smithi069 ceph-mon[53087]: pgmap v334: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 426 B/s wr, 0 op/s 2024-03-26T02:06:30.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:29 smithi019 ceph-mon[66359]: pgmap v335: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 682 B/s wr, 1 op/s 2024-03-26T02:06:30.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:29 smithi019 ceph-mon[66359]: Evicting (and blocklisting) client session 14536 (172.21.15.19:0/1467571761) 2024-03-26T02:06:30.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:29 smithi019 ceph-mon[66359]: from='mds.? [v2:172.21.15.19:6834/2546344341,v1:172.21.15.19:6835/2546344341]' entity='mds.foofs.smithi019.lmryom' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.19:0/1467571761"}]: dispatch 2024-03-26T02:06:30.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:29 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:30.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:29 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:30.335 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:29 smithi069 ceph-mon[53087]: pgmap v335: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 682 B/s wr, 1 op/s 2024-03-26T02:06:30.335 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:29 smithi069 ceph-mon[53087]: Evicting (and blocklisting) client session 14536 (172.21.15.19:0/1467571761) 2024-03-26T02:06:30.335 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:29 smithi069 ceph-mon[53087]: from='mds.? [v2:172.21.15.19:6834/2546344341,v1:172.21.15.19:6835/2546344341]' entity='mds.foofs.smithi019.lmryom' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.19:0/1467571761"}]: dispatch 2024-03-26T02:06:30.335 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:29 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:30.335 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:29 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:31.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:30 smithi019 ceph-mon[66359]: from='mds.? [v2:172.21.15.19:6834/2546344341,v1:172.21.15.19:6835/2546344341]' entity='mds.foofs.smithi019.lmryom' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.19:0/1467571761"}]': finished 2024-03-26T02:06:31.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:30 smithi019 ceph-mon[66359]: osdmap e167: 8 total, 8 up, 8 in 2024-03-26T02:06:31.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:31.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:06:31.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:30 smithi069 ceph-mon[53087]: from='mds.? [v2:172.21.15.19:6834/2546344341,v1:172.21.15.19:6835/2546344341]' entity='mds.foofs.smithi019.lmryom' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.19:0/1467571761"}]': finished 2024-03-26T02:06:31.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:30 smithi069 ceph-mon[53087]: osdmap e167: 8 total, 8 up, 8 in 2024-03-26T02:06:31.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:31.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:06:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:31 smithi069 ceph-mon[53087]: pgmap v337: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T02:06:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:31 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:31 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi069.uawdhc", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-26T02:06:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:31 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-26T02:06:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:31 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-26T02:06:32.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:31 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:06:32.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:31 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-26T02:06:32.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:31 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-26T02:06:32.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:31 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi069.uawdhc-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T02:06:32.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:31 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:06:32.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:31 smithi019 ceph-mon[66359]: pgmap v337: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T02:06:32.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:31 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:32.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:31 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi069.uawdhc", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-03-26T02:06:32.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:31 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-03-26T02:06:32.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:31 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-03-26T02:06:32.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:31 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:06:32.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:31 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-03-26T02:06:32.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:31 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-03-26T02:06:32.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:31 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi069.uawdhc-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T02:06:32.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:31 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:06:33.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:32 smithi019 ceph-mon[66359]: Upgrade: Updating nfs.foo.1.0.smithi069.uawdhc (2/2) 2024-03-26T02:06:33.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:32 smithi019 ceph-mon[66359]: Creating key for client.nfs.foo.1.0.smithi069.uawdhc 2024-03-26T02:06:33.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:32 smithi019 ceph-mon[66359]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-03-26T02:06:33.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:32 smithi019 ceph-mon[66359]: Rados config object exists: conf-nfs.foo 2024-03-26T02:06:33.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:32 smithi019 ceph-mon[66359]: Creating key for client.nfs.foo.1.0.smithi069.uawdhc-rgw 2024-03-26T02:06:33.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:32 smithi019 ceph-mon[66359]: Bind address in nfs.foo.1.0.smithi069.uawdhc's ganesha conf is defaulting to empty 2024-03-26T02:06:33.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:32 smithi019 ceph-mon[66359]: Deploying daemon nfs.foo.1.0.smithi069.uawdhc on smithi069 2024-03-26T02:06:33.270 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:33 smithi069 ceph-mon[53087]: Upgrade: Updating nfs.foo.1.0.smithi069.uawdhc (2/2) 2024-03-26T02:06:33.270 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:33 smithi069 ceph-mon[53087]: Creating key for client.nfs.foo.1.0.smithi069.uawdhc 2024-03-26T02:06:33.270 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:33 smithi069 ceph-mon[53087]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-03-26T02:06:33.270 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:33 smithi069 ceph-mon[53087]: Rados config object exists: conf-nfs.foo 2024-03-26T02:06:33.270 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:33 smithi069 ceph-mon[53087]: Creating key for client.nfs.foo.1.0.smithi069.uawdhc-rgw 2024-03-26T02:06:33.270 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:33 smithi069 ceph-mon[53087]: Bind address in nfs.foo.1.0.smithi069.uawdhc's ganesha conf is defaulting to empty 2024-03-26T02:06:33.271 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:33 smithi069 ceph-mon[53087]: Deploying daemon nfs.foo.1.0.smithi069.uawdhc on smithi069 2024-03-26T02:06:34.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:34 smithi019 ceph-mon[66359]: pgmap v338: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 921 B/s wr, 2 op/s 2024-03-26T02:06:34.266 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:34 smithi069 ceph-mon[53087]: pgmap v338: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 921 B/s wr, 2 op/s 2024-03-26T02:06:36.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:35 smithi019 ceph-mon[66359]: pgmap v339: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-26T02:06:36.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:35 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:36.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:35 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:36.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:35 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:06:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:35 smithi069 ceph-mon[53087]: pgmap v339: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-26T02:06:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:35 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:35 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:36.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:35 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:06:38.058 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:37 smithi019 ceph-mon[66359]: pgmap v340: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-26T02:06:38.058 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:37 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:38.058 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:37 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:38.058 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:37 smithi019 ceph-mon[66359]: from='mds.? [v2:172.21.15.19:6834/2546344341,v1:172.21.15.19:6835/2546344341]' entity='mds.foofs.smithi019.lmryom' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.69:0/1561493568"}]: dispatch 2024-03-26T02:06:38.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:37 smithi069 ceph-mon[53087]: pgmap v340: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-26T02:06:38.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:37 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:38.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:37 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:38.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:37 smithi069 ceph-mon[53087]: from='mds.? [v2:172.21.15.19:6834/2546344341,v1:172.21.15.19:6835/2546344341]' entity='mds.foofs.smithi019.lmryom' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.69:0/1561493568"}]: dispatch 2024-03-26T02:06:39.084 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:38 smithi019 ceph-mon[66359]: Evicting (and blocklisting) client session 24353 (172.21.15.69:0/1561493568) 2024-03-26T02:06:39.084 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:38 smithi019 ceph-mon[66359]: from='mds.? [v2:172.21.15.19:6834/2546344341,v1:172.21.15.19:6835/2546344341]' entity='mds.foofs.smithi019.lmryom' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.69:0/1561493568"}]': finished 2024-03-26T02:06:39.084 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:38 smithi019 ceph-mon[66359]: osdmap e168: 8 total, 8 up, 8 in 2024-03-26T02:06:39.084 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:38 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:39.084 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:38 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:39.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:38 smithi069 ceph-mon[53087]: Evicting (and blocklisting) client session 24353 (172.21.15.69:0/1561493568) 2024-03-26T02:06:39.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:38 smithi069 ceph-mon[53087]: from='mds.? [v2:172.21.15.19:6834/2546344341,v1:172.21.15.19:6835/2546344341]' entity='mds.foofs.smithi019.lmryom' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.69:0/1561493568"}]': finished 2024-03-26T02:06:39.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:38 smithi069 ceph-mon[53087]: osdmap e168: 8 total, 8 up, 8 in 2024-03-26T02:06:39.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:38 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:39.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:38 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:40.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:39 smithi069 ceph-mon[53087]: pgmap v342: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-03-26T02:06:40.167 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:39 smithi019 ceph-mon[66359]: pgmap v342: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-03-26T02:06:41.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:41 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:41.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:41 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:41.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:41 smithi019 ceph-mon[66359]: pgmap v343: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-26T02:06:41.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:41 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:41.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:41 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:41.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:41 smithi069 ceph-mon[53087]: pgmap v343: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-26T02:06:43.023 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:42 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:43.023 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:42 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:43.023 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:42 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:43.023 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:42 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:43.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:42 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:43.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:42 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:43.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:42 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:43.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:42 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:43.849 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:43 smithi019 ceph-mon[66359]: pgmap v344: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-03-26T02:06:44.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:43 smithi069 ceph-mon[53087]: pgmap v344: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-03-26T02:06:46.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:45 smithi069 ceph-mon[53087]: pgmap v345: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-26T02:06:46.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:46.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:46.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:06:46.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:06:46.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:46.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:46.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:06:46.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:46.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:46.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:46.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:46.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:46.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:46.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:46.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:46.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:46.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:46.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:46.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:46.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi019.qbizxo"}]: dispatch 2024-03-26T02:06:46.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi019.qbizxo"}]': finished 2024-03-26T02:06:46.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi069.uawdhc"}]: dispatch 2024-03-26T02:06:46.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi069.uawdhc"}]': finished 2024-03-26T02:06:46.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:46.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:46.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:45 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:06:46.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:45 smithi019 ceph-mon[66359]: pgmap v345: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-26T02:06:46.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:46.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:46.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:06:46.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:06:46.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:46.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:46.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:06:46.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:46.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:46.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:46.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:46.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:46.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:46.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:46.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:46.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:46.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:46.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:46.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:46.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi019.qbizxo"}]: dispatch 2024-03-26T02:06:46.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi019.qbizxo"}]': finished 2024-03-26T02:06:46.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi069.uawdhc"}]: dispatch 2024-03-26T02:06:46.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi069.uawdhc"}]': finished 2024-03-26T02:06:46.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:46.267 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:46.268 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:45 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:06:47.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:46 smithi019 ceph-mon[66359]: Upgrade: Setting container_image for all nfs 2024-03-26T02:06:47.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:46 smithi019 ceph-mon[66359]: Upgrade: Setting container_image for all nvmeof 2024-03-26T02:06:47.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:46 smithi069 ceph-mon[53087]: Upgrade: Setting container_image for all nfs 2024-03-26T02:06:47.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:46 smithi069 ceph-mon[53087]: Upgrade: Setting container_image for all nvmeof 2024-03-26T02:06:48.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:47 smithi069 ceph-mon[53087]: pgmap v346: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-26T02:06:48.129 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:47 smithi019 ceph-mon[66359]: pgmap v346: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-26T02:06:49.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:48 smithi019 ceph-mon[66359]: Upgrade: Updating node-exporter.smithi019 (1/2) 2024-03-26T02:06:49.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:48 smithi019 ceph-mon[66359]: Deploying daemon node-exporter.smithi019 on smithi019 2024-03-26T02:06:49.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:48 smithi069 ceph-mon[53087]: Upgrade: Updating node-exporter.smithi019 (1/2) 2024-03-26T02:06:49.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:48 smithi069 ceph-mon[53087]: Deploying daemon node-exporter.smithi019 on smithi019 2024-03-26T02:06:49.564 INFO:teuthology.orchestra.run.smithi019.stdout:true 2024-03-26T02:06:50.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:49 smithi069 ceph-mon[53087]: pgmap v347: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 408 B/s wr, 0 op/s 2024-03-26T02:06:50.168 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:49 smithi019 ceph-mon[66359]: pgmap v347: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 408 B/s wr, 0 op/s 2024-03-26T02:06:50.352 INFO:teuthology.orchestra.run.smithi019.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:06:50.353 INFO:teuthology.orchestra.run.smithi019.stdout:alertmanager.smithi019 smithi019 *:9093,9094 running (10m) 9s ago 20m 25.0M - 0.23.0 ba2b418f427c a6630841c2b2 2024-03-26T02:06:50.353 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi019 smithi019 running (7m) 9s ago 19m 6660k - 19.0.0-1775-g5e70124b 5f21c9264136 c5f91a47ddf2 2024-03-26T02:06:50.353 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi069 smithi069 running (6m) 12s ago 19m 6669k - 19.0.0-1775-g5e70124b 5f21c9264136 05128803b49e 2024-03-26T02:06:50.353 INFO:teuthology.orchestra.run.smithi019.stdout:grafana.smithi019 smithi019 *:3000 running (10m) 9s ago 19m 53.5M - 8.3.5 dad864ee21e9 a97dea3b5fec 2024-03-26T02:06:50.353 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi019.lmryom smithi019 running (59s) 9s ago 13m 36.0M - 19.0.0-1775-g5e70124b 5f21c9264136 499227af5537 2024-03-26T02:06:50.353 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi069.spgwgo smithi069 running (35s) 12s ago 13m 11.4M - 19.0.0-1775-g5e70124b 5f21c9264136 b8adfbbfa78e 2024-03-26T02:06:50.353 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi019.ixuvdm smithi019 *:8443,9283,8765 running (9m) 9s ago 21m 514M - 19.0.0-1775-g5e70124b 5f21c9264136 4460b22c7a56 2024-03-26T02:06:50.353 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi069.wpbewu smithi069 *:8443,9283,8765 running (8m) 12s ago 19m 455M - 19.0.0-1775-g5e70124b 5f21c9264136 380fd33842a8 2024-03-26T02:06:50.353 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi019 smithi019 running (8m) 9s ago 21m 62.9M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 c18934705774 2024-03-26T02:06:50.353 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi069 smithi069 running (8m) 12s ago 19m 47.0M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 a856bd263104 2024-03-26T02:06:50.353 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.0.0.smithi019.qbizxo smithi019 *:2049 running (21s) 9s ago 13m 61.8M - 5.7 5f21c9264136 601e946cdac2 2024-03-26T02:06:50.353 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.1.0.smithi069.uawdhc smithi069 *:2049 running (15s) 12s ago 13m 19.1M - 5.7 5f21c9264136 c8bc7245da23 2024-03-26T02:06:50.353 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi019 smithi019 *:9100 running (10m) 9s ago 19m 9751k - 1.3.1 1dbe0e931976 0d1d184bec63 2024-03-26T02:06:50.354 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi069 smithi069 *:9100 running (9m) 12s ago 19m 19.3M - 1.3.1 1dbe0e931976 c49a006f94e7 2024-03-26T02:06:50.354 INFO:teuthology.orchestra.run.smithi019.stdout:osd.0 smithi019 running (6m) 9s ago 18m 410M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 3f931cfe320a 2024-03-26T02:06:50.354 INFO:teuthology.orchestra.run.smithi019.stdout:osd.1 smithi019 running (5m) 9s ago 17m 374M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 758caf4ed841 2024-03-26T02:06:50.354 INFO:teuthology.orchestra.run.smithi019.stdout:osd.2 smithi019 running (4m) 9s ago 17m 382M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 6a9b701517ca 2024-03-26T02:06:50.354 INFO:teuthology.orchestra.run.smithi019.stdout:osd.3 smithi019 running (3m) 9s ago 16m 420M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 0401940d882b 2024-03-26T02:06:50.354 INFO:teuthology.orchestra.run.smithi019.stdout:osd.4 smithi069 running (3m) 12s ago 16m 343M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 6eb3b2ce9978 2024-03-26T02:06:50.354 INFO:teuthology.orchestra.run.smithi019.stdout:osd.5 smithi069 running (2m) 12s ago 16m 327M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 fb31f0e68fb7 2024-03-26T02:06:50.354 INFO:teuthology.orchestra.run.smithi019.stdout:osd.6 smithi069 running (2m) 12s ago 15m 301M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 924e2a35053c 2024-03-26T02:06:50.354 INFO:teuthology.orchestra.run.smithi019.stdout:osd.7 smithi069 running (81s) 12s ago 15m 287M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 d8052b0a18d6 2024-03-26T02:06:50.354 INFO:teuthology.orchestra.run.smithi019.stdout:prometheus.smithi019 smithi019 *:9095 running (8m) 9s ago 18m 69.6M - 2.33.4 514e6a882f6e 29f89db72a07 2024-03-26T02:06:50.780 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:06:50.780 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": { 2024-03-26T02:06:50.781 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:06:50.781 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:06:50.781 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": { 2024-03-26T02:06:50.781 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:06:50.781 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:06:50.781 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": { 2024-03-26T02:06:50.781 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 8 2024-03-26T02:06:50.781 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:06:50.781 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": { 2024-03-26T02:06:50.781 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:06:50.781 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:06:50.781 INFO:teuthology.orchestra.run.smithi019.stdout: "overall": { 2024-03-26T02:06:50.781 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 14 2024-03-26T02:06:50.781 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-26T02:06:50.781 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:06:51.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:50 smithi069 ceph-mon[53087]: from='client.34550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:06:51.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:50 smithi069 ceph-mon[53087]: from='client.34554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:06:51.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:50 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/2488104536' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:51.148 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:06:51.149 INFO:teuthology.orchestra.run.smithi019.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2", 2024-03-26T02:06:51.149 INFO:teuthology.orchestra.run.smithi019.stdout: "in_progress": true, 2024-03-26T02:06:51.149 INFO:teuthology.orchestra.run.smithi019.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T02:06:51.149 INFO:teuthology.orchestra.run.smithi019.stdout: "services_complete": [ 2024-03-26T02:06:51.149 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr", 2024-03-26T02:06:51.149 INFO:teuthology.orchestra.run.smithi019.stdout: "mds", 2024-03-26T02:06:51.149 INFO:teuthology.orchestra.run.smithi019.stdout: "crash", 2024-03-26T02:06:51.149 INFO:teuthology.orchestra.run.smithi019.stdout: "mon", 2024-03-26T02:06:51.149 INFO:teuthology.orchestra.run.smithi019.stdout: "nfs", 2024-03-26T02:06:51.149 INFO:teuthology.orchestra.run.smithi019.stdout: "osd" 2024-03-26T02:06:51.149 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-26T02:06:51.149 INFO:teuthology.orchestra.run.smithi019.stdout: "progress": "18/23 daemons upgraded", 2024-03-26T02:06:51.149 INFO:teuthology.orchestra.run.smithi019.stdout: "message": "Currently upgrading node-exporter daemons", 2024-03-26T02:06:51.150 INFO:teuthology.orchestra.run.smithi019.stdout: "is_paused": false 2024-03-26T02:06:51.150 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:06:51.150 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:50 smithi019 ceph-mon[66359]: from='client.34550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:06:51.150 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:50 smithi019 ceph-mon[66359]: from='client.34554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:06:51.150 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:50 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/2488104536' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:06:51.679 INFO:teuthology.orchestra.run.smithi019.stdout:HEALTH_OK 2024-03-26T02:06:52.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:51 smithi019 ceph-mon[66359]: from='client.34558 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:06:52.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:51 smithi019 ceph-mon[66359]: pgmap v348: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-03-26T02:06:52.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:51 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/968614537' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:06:52.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:51 smithi069 ceph-mon[53087]: from='client.34558 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:06:52.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:51 smithi069 ceph-mon[53087]: pgmap v348: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-03-26T02:06:52.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:51 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/968614537' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:06:53.247 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:52 smithi019 ceph-mon[66359]: from='client.34566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:06:53.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:52 smithi069 ceph-mon[53087]: from='client.34566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:06:54.194 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:53 smithi069 ceph-mon[53087]: pgmap v349: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-03-26T02:06:54.194 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:53 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:54.194 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:53 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:54.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:53 smithi019 ceph-mon[66359]: pgmap v349: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-03-26T02:06:54.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:53 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:54.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:53 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:06:56.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:55 smithi019 ceph-mon[66359]: pgmap v350: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:06:56.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:55 smithi019 ceph-mon[66359]: Upgrade: Updating node-exporter.smithi069 (2/2) 2024-03-26T02:06:56.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:55 smithi019 ceph-mon[66359]: Deploying daemon node-exporter.smithi069 on smithi069 2024-03-26T02:06:56.268 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:55 smithi069 ceph-mon[53087]: pgmap v350: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:06:56.268 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:55 smithi069 ceph-mon[53087]: Upgrade: Updating node-exporter.smithi069 (2/2) 2024-03-26T02:06:56.268 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:55 smithi069 ceph-mon[53087]: Deploying daemon node-exporter.smithi069 on smithi069 2024-03-26T02:06:58.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:57 smithi019 ceph-mon[66359]: pgmap v351: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:06:58.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:57 smithi069 ceph-mon[53087]: pgmap v351: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:07:00.038 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:06:59 smithi069 ceph-mon[53087]: pgmap v352: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:07:00.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:06:59 smithi019 ceph-mon[66359]: pgmap v352: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:07:01.252 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:01.252 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:01.252 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:07:01.253 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:07:01.253 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:01 smithi069 ceph-mon[53087]: pgmap v353: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 167 B/s rd, 0 op/s 2024-03-26T02:07:01.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:01.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:01.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:07:01.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:07:01.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:01 smithi019 ceph-mon[66359]: pgmap v353: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 167 B/s rd, 0 op/s 2024-03-26T02:07:04.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:04 smithi019 ceph-mon[66359]: pgmap v354: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 166 B/s rd, 0 op/s 2024-03-26T02:07:04.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:04 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:04.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:04 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:04.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:04 smithi069 ceph-mon[53087]: pgmap v354: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 166 B/s rd, 0 op/s 2024-03-26T02:07:04.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:04 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:04.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:04 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:05.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:05.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:05.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:05 smithi019 ceph-mon[66359]: pgmap v355: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 250 B/s rd, 0 op/s 2024-03-26T02:07:05.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:05.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:05.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:05 smithi069 ceph-mon[53087]: pgmap v355: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 250 B/s rd, 0 op/s 2024-03-26T02:07:07.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:06 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:07.017 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:06 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:07.095 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:06 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:07.096 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:06 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:08.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:07 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:08.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:07 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:08.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:07 smithi019 ceph-mon[66359]: pgmap v356: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 166 B/s rd, 0 op/s 2024-03-26T02:07:08.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:07 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:08.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:07 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:08.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:07 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:08.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:07 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:08.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:07 smithi069 ceph-mon[53087]: pgmap v356: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 166 B/s rd, 0 op/s 2024-03-26T02:07:08.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:07 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:08.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:07 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:10.397 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:10 smithi019 ceph-mon[66359]: pgmap v357: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 250 B/s rd, 0 op/s 2024-03-26T02:07:10.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:10 smithi069 ceph-mon[53087]: pgmap v357: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 250 B/s rd, 0 op/s 2024-03-26T02:07:11.920 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:11 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:11.955 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:11 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:11.956 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:11 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:07:11.956 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:11 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:07:11.956 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:11 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:11.956 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:11 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:11.956 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:11 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:07:11.956 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:11 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:11.956 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:11 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:11.956 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:11 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:11.956 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:11 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:11.956 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:11 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:11.956 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:11 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:11.957 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:11 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:11.957 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:11 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:11.957 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:11 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:11.957 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:11 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:11.957 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:11 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:11.957 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:11 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:11.957 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:11 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:11.957 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:11 smithi019 ceph-mon[66359]: pgmap v358: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 166 B/s rd, 0 op/s 2024-03-26T02:07:12.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:11 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:12.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:11 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:12.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:11 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:07:12.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:11 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:07:12.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:11 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:12.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:11 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:12.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:11 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:07:12.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:11 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:12.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:11 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:12.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:11 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:12.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:11 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:12.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:11 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:12.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:11 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:12.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:11 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:12.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:11 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:12.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:11 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:12.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:11 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:12.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:11 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:12.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:11 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:12.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:11 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:12.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:11 smithi069 ceph-mon[53087]: pgmap v358: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 166 B/s rd, 0 op/s 2024-03-26T02:07:14.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:14 smithi019 ceph-mon[66359]: Upgrade: Updating prometheus.smithi019 2024-03-26T02:07:14.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:14 smithi019 ceph-mon[66359]: Deploying daemon prometheus.smithi019 on smithi019 2024-03-26T02:07:14.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:14 smithi019 ceph-mon[66359]: pgmap v359: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:07:14.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:14 smithi069 ceph-mon[53087]: Upgrade: Updating prometheus.smithi019 2024-03-26T02:07:14.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:14 smithi069 ceph-mon[53087]: Deploying daemon prometheus.smithi019 on smithi019 2024-03-26T02:07:14.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:14 smithi069 ceph-mon[53087]: pgmap v359: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:07:16.245 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:16 smithi019 ceph-mon[66359]: pgmap v360: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:07:16.245 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:16 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:07:16.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:16 smithi069 ceph-mon[53087]: pgmap v360: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:07:16.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:16 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:07:18.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:18 smithi019 ceph-mon[66359]: pgmap v361: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:07:18.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:18 smithi069 ceph-mon[53087]: pgmap v361: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:07:19.977 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:19 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:19.978 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:19 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:19.978 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:19 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:07:19.978 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:19 smithi019 ceph-mon[66359]: pgmap v362: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:07:19.978 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:19 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:19.978 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:19 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:20.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:19 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:20.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:19 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:20.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:19 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:07:20.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:19 smithi069 ceph-mon[53087]: pgmap v362: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:07:20.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:19 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:20.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:19 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:22.041 INFO:teuthology.orchestra.run.smithi019.stdout:true 2024-03-26T02:07:22.344 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:22 smithi019 ceph-mon[66359]: pgmap v363: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:07:22.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:22 smithi069 ceph-mon[53087]: pgmap v363: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:07:22.776 INFO:teuthology.orchestra.run.smithi019.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:07:22.777 INFO:teuthology.orchestra.run.smithi019.stdout:alertmanager.smithi019 smithi019 *:9093,9094 running (11m) 17s ago 20m 25.1M - 0.23.0 ba2b418f427c a6630841c2b2 2024-03-26T02:07:22.777 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi019 smithi019 running (7m) 17s ago 20m 6660k - 19.0.0-1775-g5e70124b 5f21c9264136 c5f91a47ddf2 2024-03-26T02:07:22.777 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi069 smithi069 running (7m) 19s ago 19m 6669k - 19.0.0-1775-g5e70124b 5f21c9264136 05128803b49e 2024-03-26T02:07:22.777 INFO:teuthology.orchestra.run.smithi019.stdout:grafana.smithi019 smithi019 *:3000 running (10m) 17s ago 20m 54.8M - 8.3.5 dad864ee21e9 a97dea3b5fec 2024-03-26T02:07:22.777 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi019.lmryom smithi019 running (91s) 17s ago 14m 36.3M - 19.0.0-1775-g5e70124b 5f21c9264136 499227af5537 2024-03-26T02:07:22.777 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi069.spgwgo smithi069 running (68s) 19s ago 14m 11.5M - 19.0.0-1775-g5e70124b 5f21c9264136 b8adfbbfa78e 2024-03-26T02:07:22.777 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi019.ixuvdm smithi019 *:8443,9283,8765 running (10m) 17s ago 21m 515M - 19.0.0-1775-g5e70124b 5f21c9264136 4460b22c7a56 2024-03-26T02:07:22.777 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi069.wpbewu smithi069 *:8443,9283,8765 running (9m) 19s ago 19m 455M - 19.0.0-1775-g5e70124b 5f21c9264136 380fd33842a8 2024-03-26T02:07:22.777 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi019 smithi019 running (9m) 17s ago 21m 64.0M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 c18934705774 2024-03-26T02:07:22.777 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi069 smithi069 running (8m) 19s ago 19m 48.1M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 a856bd263104 2024-03-26T02:07:22.777 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.0.0.smithi019.qbizxo smithi019 *:2049 running (53s) 17s ago 14m 62.0M - 5.7 5f21c9264136 601e946cdac2 2024-03-26T02:07:22.777 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.1.0.smithi069.uawdhc smithi069 *:2049 running (48s) 19s ago 14m 61.6M - 5.7 5f21c9264136 c8bc7245da23 2024-03-26T02:07:22.777 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi019 smithi019 *:9100 running (29s) 17s ago 20m 7868k - 1.5.0 0da6a335fe13 aac97ec99f0a 2024-03-26T02:07:22.778 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi069 smithi069 *:9100 running (23s) 19s ago 19m 3590k - 1.5.0 0da6a335fe13 6d0c5a6932ef 2024-03-26T02:07:22.778 INFO:teuthology.orchestra.run.smithi019.stdout:osd.0 smithi019 running (6m) 17s ago 18m 411M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 3f931cfe320a 2024-03-26T02:07:22.778 INFO:teuthology.orchestra.run.smithi019.stdout:osd.1 smithi019 running (6m) 17s ago 18m 374M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 758caf4ed841 2024-03-26T02:07:22.778 INFO:teuthology.orchestra.run.smithi019.stdout:osd.2 smithi019 running (5m) 17s ago 17m 382M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 6a9b701517ca 2024-03-26T02:07:22.778 INFO:teuthology.orchestra.run.smithi019.stdout:osd.3 smithi019 running (4m) 17s ago 17m 421M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 0401940d882b 2024-03-26T02:07:22.778 INFO:teuthology.orchestra.run.smithi019.stdout:osd.4 smithi069 running (3m) 19s ago 17m 344M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 6eb3b2ce9978 2024-03-26T02:07:22.778 INFO:teuthology.orchestra.run.smithi019.stdout:osd.5 smithi069 running (3m) 19s ago 16m 327M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 fb31f0e68fb7 2024-03-26T02:07:22.778 INFO:teuthology.orchestra.run.smithi019.stdout:osd.6 smithi069 running (2m) 19s ago 16m 302M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 924e2a35053c 2024-03-26T02:07:22.778 INFO:teuthology.orchestra.run.smithi019.stdout:osd.7 smithi069 running (113s) 19s ago 15m 287M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 d8052b0a18d6 2024-03-26T02:07:22.778 INFO:teuthology.orchestra.run.smithi019.stdout:prometheus.smithi019 smithi019 *:9095 starting - - - - 2024-03-26T02:07:23.221 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:07:23.221 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": { 2024-03-26T02:07:23.222 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:07:23.222 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:07:23.222 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": { 2024-03-26T02:07:23.222 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:07:23.222 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:07:23.222 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": { 2024-03-26T02:07:23.222 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 8 2024-03-26T02:07:23.222 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:07:23.222 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": { 2024-03-26T02:07:23.222 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:07:23.222 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:07:23.222 INFO:teuthology.orchestra.run.smithi019.stdout: "overall": { 2024-03-26T02:07:23.223 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 14 2024-03-26T02:07:23.223 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-26T02:07:23.223 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:07:23.315 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:23 smithi019 ceph-mon[66359]: from='client.34574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:07:23.588 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:07:23.588 INFO:teuthology.orchestra.run.smithi019.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2", 2024-03-26T02:07:23.588 INFO:teuthology.orchestra.run.smithi019.stdout: "in_progress": true, 2024-03-26T02:07:23.588 INFO:teuthology.orchestra.run.smithi019.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T02:07:23.588 INFO:teuthology.orchestra.run.smithi019.stdout: "services_complete": [ 2024-03-26T02:07:23.588 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr", 2024-03-26T02:07:23.588 INFO:teuthology.orchestra.run.smithi019.stdout: "mds", 2024-03-26T02:07:23.589 INFO:teuthology.orchestra.run.smithi019.stdout: "crash", 2024-03-26T02:07:23.589 INFO:teuthology.orchestra.run.smithi019.stdout: "mon", 2024-03-26T02:07:23.589 INFO:teuthology.orchestra.run.smithi019.stdout: "nfs", 2024-03-26T02:07:23.589 INFO:teuthology.orchestra.run.smithi019.stdout: "osd" 2024-03-26T02:07:23.589 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-26T02:07:23.589 INFO:teuthology.orchestra.run.smithi019.stdout: "progress": "18/23 daemons upgraded", 2024-03-26T02:07:23.589 INFO:teuthology.orchestra.run.smithi019.stdout: "message": "Currently upgrading prometheus daemons", 2024-03-26T02:07:23.589 INFO:teuthology.orchestra.run.smithi019.stdout: "is_paused": false 2024-03-26T02:07:23.589 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:07:23.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:23 smithi069 ceph-mon[53087]: from='client.34574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:07:24.028 INFO:teuthology.orchestra.run.smithi019.stdout:HEALTH_OK 2024-03-26T02:07:24.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:24 smithi019 ceph-mon[66359]: from='client.34578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:07:24.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:24 smithi019 ceph-mon[66359]: from='client.34582 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:07:24.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:24 smithi019 ceph-mon[66359]: pgmap v364: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:07:24.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:24 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/4187916155' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:24.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:24 smithi019 ceph-mon[66359]: from='client.34590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:07:24.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:24 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/1645572610' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:07:24.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:24 smithi069 ceph-mon[53087]: from='client.34578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:07:24.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:24 smithi069 ceph-mon[53087]: from='client.34582 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:07:24.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:24 smithi069 ceph-mon[53087]: pgmap v364: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:07:24.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:24 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/4187916155' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:24.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:24 smithi069 ceph-mon[53087]: from='client.34590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:07:24.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:24 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/1645572610' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:07:25.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:25 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:25.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:25 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:25.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:25 smithi019 ceph-mon[66359]: pgmap v365: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:07:25.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:25 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:25.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:25 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:25.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:25 smithi069 ceph-mon[53087]: pgmap v365: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:07:26.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:26 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:26.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:26 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:27.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:26 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:27.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:26 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:27.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:27 smithi069 ceph-mon[53087]: pgmap v366: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:07:28.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:27 smithi019 ceph-mon[66359]: pgmap v366: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:07:30.200 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:30 smithi019 ceph-mon[66359]: pgmap v367: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:07:30.200 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:30.200 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:30.200 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:07:30.200 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:07:30.200 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:30.200 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:30.200 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:07:30.200 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:07:30.200 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:30.201 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:30.201 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:30.201 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:30.201 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:30.201 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:30.201 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:30.201 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:30.201 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:30.201 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:30.201 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:30.201 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:30.201 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:30.201 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:30.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:30 smithi069 ceph-mon[53087]: pgmap v367: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:07:30.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:30.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:30.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:07:30.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:07:30.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:30.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:30.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:07:30.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:07:30.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:30.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:30.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:30.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:30.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:30.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:30.611 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:30.611 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:30.611 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:30.611 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:30.611 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:30.611 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:30.611 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:30.611 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:31.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:31 smithi019 ceph-mon[66359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:07:31.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:31 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:07:31.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:31 smithi069 ceph-mon[53087]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:07:31.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:31 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:07:32.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:32 smithi019 ceph-mon[66359]: Upgrade: Updating alertmanager.smithi019 2024-03-26T02:07:32.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:32 smithi019 ceph-mon[66359]: Deploying daemon alertmanager.smithi019 on smithi019 2024-03-26T02:07:32.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:32 smithi019 ceph-mon[66359]: pgmap v368: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:07:32.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:32 smithi069 ceph-mon[53087]: Upgrade: Updating alertmanager.smithi019 2024-03-26T02:07:32.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:32 smithi069 ceph-mon[53087]: Deploying daemon alertmanager.smithi019 on smithi019 2024-03-26T02:07:32.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:32 smithi069 ceph-mon[53087]: pgmap v368: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:07:33.438 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:33 smithi019 ceph-mon[66359]: pgmap v369: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:07:33.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:33 smithi069 ceph-mon[53087]: pgmap v369: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:07:36.236 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:36 smithi019 ceph-mon[66359]: pgmap v370: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:07:36.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:36 smithi069 ceph-mon[53087]: pgmap v370: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:07:38.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:38 smithi019 ceph-mon[66359]: pgmap v371: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:07:38.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:38 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:38.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:38 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:38.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:38 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:07:38.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:38 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:38.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:38 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:38.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:38 smithi069 ceph-mon[53087]: pgmap v371: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:07:38.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:38 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:38.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:38 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:38.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:38 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:07:38.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:38 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:38.610 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:38 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:39.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:39 smithi019 ceph-mon[66359]: pgmap v372: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:07:39.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:39 smithi069 ceph-mon[53087]: pgmap v372: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:07:42.225 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:42 smithi019 ceph-mon[66359]: pgmap v373: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:07:42.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:42 smithi069 ceph-mon[53087]: pgmap v373: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:07:43.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:43 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:43.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:43 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:43.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:43 smithi019 ceph-mon[66359]: pgmap v374: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:07:43.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:43 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:43.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:43 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:43.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:43 smithi069 ceph-mon[53087]: pgmap v374: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:07:45.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:44 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:45.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:44 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:45.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:44 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:45.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:44 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:45.872 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:45 smithi019 ceph-mon[66359]: pgmap v375: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:07:46.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:45 smithi069 ceph-mon[53087]: pgmap v375: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:07:46.978 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:46 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:07:47.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:46 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:07:47.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:47 smithi019 ceph-mon[66359]: pgmap v376: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:07:47.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:47 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:47.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:47 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:47.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:47 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:07:47.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:47 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:07:47.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:47 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:47.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:47 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:47.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:47 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T02:07:48.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:47 smithi069 ceph-mon[53087]: pgmap v376: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:07:48.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:47 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:48.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:47 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:48.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:47 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:07:48.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:47 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:07:48.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:47 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:48.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:47 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:48.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:47 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T02:07:49.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:48 smithi019 ceph-mon[66359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T02:07:49.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:48 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:07:49.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:48 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:49.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:48 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:49.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:48 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:49.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:48 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:49.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:48 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:49.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:48 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:49.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:48 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:49.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:48 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:49.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:48 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:49.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:48 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:49.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:48 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:49.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:48 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:49.016 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:48 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:49.017 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:48 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:49.017 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:48 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:49.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:48 smithi069 ceph-mon[53087]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T02:07:49.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:48 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:07:49.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:48 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:49.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:48 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:49.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:48 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:49.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:48 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:49.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:48 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:49.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:48 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:49.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:48 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:49.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:48 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:49.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:48 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:49.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:48 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:49.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:48 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:49.110 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:48 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:49.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:48 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:49.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:48 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:49.111 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:48 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:49.927 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:49 smithi019 ceph-mon[66359]: pgmap v377: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:07:50.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:49 smithi069 ceph-mon[53087]: pgmap v377: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:07:50.952 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:50 smithi019 ceph-mon[66359]: Upgrade: Updating grafana.smithi019 2024-03-26T02:07:50.953 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:50 smithi019 ceph-mon[66359]: Deploying daemon grafana.smithi019 on smithi019 2024-03-26T02:07:51.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:50 smithi069 ceph-mon[53087]: Upgrade: Updating grafana.smithi019 2024-03-26T02:07:51.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:50 smithi069 ceph-mon[53087]: Deploying daemon grafana.smithi019 on smithi019 2024-03-26T02:07:52.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:51 smithi019 ceph-mon[66359]: pgmap v378: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:07:52.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:51 smithi069 ceph-mon[53087]: pgmap v378: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:07:54.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:54 smithi019 ceph-mon[66359]: pgmap v379: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:07:54.461 INFO:teuthology.orchestra.run.smithi019.stdout:true 2024-03-26T02:07:54.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:54 smithi069 ceph-mon[53087]: pgmap v379: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:07:55.353 INFO:teuthology.orchestra.run.smithi019.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:07:55.353 INFO:teuthology.orchestra.run.smithi019.stdout:alertmanager.smithi019 smithi019 *:9093,9094 running (18s) 12s ago 21m 15.2M - 0.25.0 c8568f914cd2 2cb7fa4e7f1c 2024-03-26T02:07:55.353 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi019 smithi019 running (8m) 12s ago 21m 6660k - 19.0.0-1775-g5e70124b 5f21c9264136 c5f91a47ddf2 2024-03-26T02:07:55.353 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi069 smithi069 running (8m) 52s ago 20m 6669k - 19.0.0-1775-g5e70124b 5f21c9264136 05128803b49e 2024-03-26T02:07:55.354 INFO:teuthology.orchestra.run.smithi019.stdout:grafana.smithi019 smithi019 *:3000 starting - - - - 2024-03-26T02:07:55.354 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi019.lmryom smithi019 running (2m) 12s ago 15m 36.5M - 19.0.0-1775-g5e70124b 5f21c9264136 499227af5537 2024-03-26T02:07:55.354 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi069.spgwgo smithi069 running (100s) 52s ago 14m 11.5M - 19.0.0-1775-g5e70124b 5f21c9264136 b8adfbbfa78e 2024-03-26T02:07:55.354 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi019.ixuvdm smithi019 *:8443,9283,8765 running (10m) 12s ago 22m 517M - 19.0.0-1775-g5e70124b 5f21c9264136 4460b22c7a56 2024-03-26T02:07:55.354 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi069.wpbewu smithi069 *:8443,9283,8765 running (9m) 52s ago 20m 455M - 19.0.0-1775-g5e70124b 5f21c9264136 380fd33842a8 2024-03-26T02:07:55.354 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi019 smithi019 running (9m) 12s ago 22m 67.9M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 c18934705774 2024-03-26T02:07:55.354 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi069 smithi069 running (9m) 52s ago 20m 48.1M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 a856bd263104 2024-03-26T02:07:55.354 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.0.0.smithi019.qbizxo smithi019 *:2049 running (86s) 12s ago 14m 63.6M - 5.7 5f21c9264136 601e946cdac2 2024-03-26T02:07:55.354 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.1.0.smithi069.uawdhc smithi069 *:2049 running (80s) 52s ago 14m 61.6M - 5.7 5f21c9264136 c8bc7245da23 2024-03-26T02:07:55.354 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi019 smithi019 *:9100 running (62s) 12s ago 20m 7977k - 1.5.0 0da6a335fe13 aac97ec99f0a 2024-03-26T02:07:55.355 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi069 smithi069 *:9100 running (55s) 52s ago 20m 3590k - 1.5.0 0da6a335fe13 6d0c5a6932ef 2024-03-26T02:07:55.355 INFO:teuthology.orchestra.run.smithi019.stdout:osd.0 smithi019 running (7m) 12s ago 19m 410M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 3f931cfe320a 2024-03-26T02:07:55.355 INFO:teuthology.orchestra.run.smithi019.stdout:osd.1 smithi019 running (6m) 12s ago 18m 375M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 758caf4ed841 2024-03-26T02:07:55.355 INFO:teuthology.orchestra.run.smithi019.stdout:osd.2 smithi019 running (5m) 12s ago 18m 383M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 6a9b701517ca 2024-03-26T02:07:55.355 INFO:teuthology.orchestra.run.smithi019.stdout:osd.3 smithi019 running (5m) 12s ago 18m 423M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 0401940d882b 2024-03-26T02:07:55.355 INFO:teuthology.orchestra.run.smithi019.stdout:osd.4 smithi069 running (4m) 52s ago 17m 344M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 6eb3b2ce9978 2024-03-26T02:07:55.355 INFO:teuthology.orchestra.run.smithi019.stdout:osd.5 smithi069 running (3m) 52s ago 17m 327M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 fb31f0e68fb7 2024-03-26T02:07:55.355 INFO:teuthology.orchestra.run.smithi019.stdout:osd.6 smithi069 running (3m) 52s ago 16m 302M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 924e2a35053c 2024-03-26T02:07:55.355 INFO:teuthology.orchestra.run.smithi019.stdout:osd.7 smithi069 running (2m) 52s ago 16m 287M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 d8052b0a18d6 2024-03-26T02:07:55.355 INFO:teuthology.orchestra.run.smithi019.stdout:prometheus.smithi019 smithi019 *:9095 running (37s) 12s ago 19m 46.2M - 2.43.0 a07b618ecd1d 86364cd8e9cf 2024-03-26T02:07:55.660 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:55 smithi019 ceph-mon[66359]: from='client.34598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:07:55.660 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:55 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:55.660 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:55 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:55.660 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:55 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:07:55.660 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:55 smithi019 ceph-mon[66359]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:07:55.660 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:55 smithi019 ceph-mon[66359]: pgmap v380: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:07:55.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:55 smithi069 ceph-mon[53087]: from='client.34598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:07:55.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:55 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:55.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:55 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:07:55.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:55 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:07:55.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:55 smithi069 ceph-mon[53087]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:07:55.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:55 smithi069 ceph-mon[53087]: pgmap v380: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:07:55.884 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:07:55.885 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": { 2024-03-26T02:07:55.885 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:07:55.885 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:07:55.885 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": { 2024-03-26T02:07:55.885 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:07:55.885 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:07:55.885 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": { 2024-03-26T02:07:55.885 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 8 2024-03-26T02:07:55.885 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:07:55.885 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": { 2024-03-26T02:07:55.885 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:07:55.885 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:07:55.886 INFO:teuthology.orchestra.run.smithi019.stdout: "overall": { 2024-03-26T02:07:55.886 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 14 2024-03-26T02:07:55.886 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-26T02:07:55.886 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:07:56.253 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:07:56.253 INFO:teuthology.orchestra.run.smithi019.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2", 2024-03-26T02:07:56.253 INFO:teuthology.orchestra.run.smithi019.stdout: "in_progress": true, 2024-03-26T02:07:56.253 INFO:teuthology.orchestra.run.smithi019.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T02:07:56.253 INFO:teuthology.orchestra.run.smithi019.stdout: "services_complete": [ 2024-03-26T02:07:56.253 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr", 2024-03-26T02:07:56.253 INFO:teuthology.orchestra.run.smithi019.stdout: "mds", 2024-03-26T02:07:56.254 INFO:teuthology.orchestra.run.smithi019.stdout: "crash", 2024-03-26T02:07:56.254 INFO:teuthology.orchestra.run.smithi019.stdout: "mon", 2024-03-26T02:07:56.254 INFO:teuthology.orchestra.run.smithi019.stdout: "nfs", 2024-03-26T02:07:56.254 INFO:teuthology.orchestra.run.smithi019.stdout: "osd" 2024-03-26T02:07:56.254 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-26T02:07:56.254 INFO:teuthology.orchestra.run.smithi019.stdout: "progress": "18/23 daemons upgraded", 2024-03-26T02:07:56.254 INFO:teuthology.orchestra.run.smithi019.stdout: "message": "Currently upgrading grafana daemons", 2024-03-26T02:07:56.254 INFO:teuthology.orchestra.run.smithi019.stdout: "is_paused": false 2024-03-26T02:07:56.254 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:07:56.681 INFO:teuthology.orchestra.run.smithi019.stdout:HEALTH_OK 2024-03-26T02:07:56.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:56 smithi019 ceph-mon[66359]: from='client.34606 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:07:56.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:56 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/4075562079' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:56.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:56 smithi069 ceph-mon[53087]: from='client.34606 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:07:56.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:56 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/4075562079' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:07:57.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:57 smithi019 ceph-mon[66359]: from='client.34614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:07:57.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:57 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/3873623520' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:07:57.765 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:07:57 smithi019 ceph-mon[66359]: pgmap v381: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:07:57.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:57 smithi069 ceph-mon[53087]: from='client.34614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:07:57.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:57 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/3873623520' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:07:57.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:07:57 smithi069 ceph-mon[53087]: pgmap v381: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:00.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:00 smithi019 ceph-mon[66359]: pgmap v382: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:08:00.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:00 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:08:00.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:00 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:08:00.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:00 smithi069 ceph-mon[53087]: pgmap v382: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:08:00.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:00 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:08:00.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:00 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:08:01.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:08:01.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:08:01.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:08:01.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:08:01.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:08:01.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:08:02.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:02 smithi019 ceph-mon[66359]: pgmap v383: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:02.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:02 smithi069 ceph-mon[53087]: pgmap v383: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:03.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:03 smithi019 ceph-mon[66359]: pgmap v384: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:03.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:03 smithi069 ceph-mon[53087]: pgmap v384: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:05.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:08:05.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:08:05.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:05.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:05.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:08:05.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:08:05.766 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T02:08:05.767 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T02:08:05.767 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:05.767 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.767 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.767 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.767 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.767 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.767 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.767 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.767 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.767 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.767 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.767 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.768 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.768 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.768 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.768 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.768 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.768 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.768 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.768 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: Upgrade: Finalizing container_image settings 2024-03-26T02:08:05.768 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:08:05.768 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-26T02:08:05.768 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-26T02:08:05.768 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:08:05.768 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-26T02:08:05.768 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-26T02:08:05.768 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-26T02:08:05.769 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-26T02:08:05.769 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-26T02:08:05.769 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-26T02:08:05.769 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-26T02:08:05.769 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-26T02:08:05.769 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-26T02:08:05.769 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-26T02:08:05.769 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-26T02:08:05.769 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:08:05.769 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-26T02:08:05.769 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-26T02:08:05.769 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-26T02:08:05.769 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-03-26T02:08:05.769 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-26T02:08:05.770 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-26T02:08:05.770 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-26T02:08:05.770 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-26T02:08:05.770 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:08:05.770 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:08:05.770 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:08:05.770 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:08:05.770 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:08:05.770 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:08:05.770 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: Upgrade: Complete! 2024-03-26T02:08:05.770 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-26T02:08:05.770 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-26T02:08:05.770 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:05.771 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: pgmap v385: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:08:05.771 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:05.771 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:05.771 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:08:05.771 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:08:05.771 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:05.771 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:05.771 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:05.771 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:08:05.771 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:05 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:08:05.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:08:05.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:08:05.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:05.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:05.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:08:05.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:08:05.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T02:08:05.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T02:08:05.860 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:05.861 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.861 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.861 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.861 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.861 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.861 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.861 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.861 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.861 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.861 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.861 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.861 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.861 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.861 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.862 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.862 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.862 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.862 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:05.862 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: Upgrade: Finalizing container_image settings 2024-03-26T02:08:05.862 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:08:05.862 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-26T02:08:05.862 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-26T02:08:05.862 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:08:05.862 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-26T02:08:05.862 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-26T02:08:05.862 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-26T02:08:05.862 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-26T02:08:05.863 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-26T02:08:05.863 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-26T02:08:05.863 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-26T02:08:05.863 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-26T02:08:05.863 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-26T02:08:05.863 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-26T02:08:05.863 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-26T02:08:05.863 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:08:05.863 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-26T02:08:05.863 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-26T02:08:05.863 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-26T02:08:05.863 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-03-26T02:08:05.863 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-26T02:08:05.863 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-26T02:08:05.864 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-26T02:08:05.864 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-26T02:08:05.864 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:08:05.864 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:08:05.864 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:08:05.864 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:08:05.864 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:08:05.864 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:08:05.864 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: Upgrade: Complete! 2024-03-26T02:08:05.864 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-26T02:08:05.864 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-26T02:08:05.864 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:05.864 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: pgmap v385: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:08:05.864 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:05.865 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:05.865 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:08:05.865 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:08:05.865 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:05.865 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:05.865 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:05.865 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:08:05.865 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:05 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:08:07.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:07 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:08:07.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:07 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:08:08.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:08 smithi069 ceph-mon[53087]: pgmap v386: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:08.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:08 smithi019 ceph-mon[66359]: pgmap v386: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:10.518 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:10 smithi019 ceph-mon[66359]: pgmap v387: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:08:10.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:10 smithi069 ceph-mon[53087]: pgmap v387: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:08:11.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:11 smithi019 ceph-mon[66359]: pgmap v388: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:11.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:11 smithi069 ceph-mon[53087]: pgmap v388: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:13.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:13 smithi019 ceph-mon[66359]: pgmap v389: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:13.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:13 smithi069 ceph-mon[53087]: pgmap v389: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:15.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:15 smithi019 ceph-mon[66359]: pgmap v390: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:08:15.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:15 smithi069 ceph-mon[53087]: pgmap v390: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:08:16.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:16 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:08:16.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:16 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:08:17.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:17 smithi019 ceph-mon[66359]: pgmap v391: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:17.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:17 smithi069 ceph-mon[53087]: pgmap v391: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:19.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:19 smithi019 ceph-mon[66359]: pgmap v392: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:08:19.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:19 smithi069 ceph-mon[53087]: pgmap v392: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:08:21.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:21 smithi019 ceph-mon[66359]: pgmap v393: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:21.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:21 smithi069 ceph-mon[53087]: pgmap v393: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:23.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:23 smithi019 ceph-mon[66359]: pgmap v394: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:23.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:23 smithi069 ceph-mon[53087]: pgmap v394: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:25.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:25 smithi019 ceph-mon[66359]: pgmap v395: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:08:25.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:25 smithi069 ceph-mon[53087]: pgmap v395: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:08:27.404 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:27 smithi019 ceph-mon[66359]: from='client.34622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:08:27.404 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:27 smithi019 ceph-mon[66359]: pgmap v396: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:27.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:27 smithi069 ceph-mon[53087]: from='client.34622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:08:27.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:27 smithi069 ceph-mon[53087]: pgmap v396: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:27.777 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'ceph orch ps' 2024-03-26T02:08:29.397 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:29 smithi019 ceph-mon[66359]: pgmap v397: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:08:29.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:29 smithi069 ceph-mon[53087]: pgmap v397: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:08:29.863 INFO:teuthology.orchestra.run.smithi019.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:08:29.864 INFO:teuthology.orchestra.run.smithi019.stdout:alertmanager.smithi019 smithi019 *:9093,9094 running (53s) 30s ago 21m 15.2M - 0.25.0 c8568f914cd2 2cb7fa4e7f1c 2024-03-26T02:08:29.864 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi019 smithi019 running (8m) 30s ago 21m 6660k - 19.0.0-1775-g5e70124b 5f21c9264136 c5f91a47ddf2 2024-03-26T02:08:29.864 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi069 smithi069 running (8m) 86s ago 21m 6669k - 19.0.0-1775-g5e70124b 5f21c9264136 05128803b49e 2024-03-26T02:08:29.864 INFO:teuthology.orchestra.run.smithi019.stdout:grafana.smithi019 smithi019 *:3000 running (36s) 30s ago 21m 51.5M - 9.4.12 a0ea5e3c46d9 cdb65c966500 2024-03-26T02:08:29.864 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi019.lmryom smithi019 running (2m) 30s ago 15m 36.7M - 19.0.0-1775-g5e70124b 5f21c9264136 499227af5537 2024-03-26T02:08:29.864 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi069.spgwgo smithi069 running (2m) 86s ago 15m 11.5M - 19.0.0-1775-g5e70124b 5f21c9264136 b8adfbbfa78e 2024-03-26T02:08:29.864 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi019.ixuvdm smithi019 *:8443,9283,8765 running (11m) 30s ago 22m 517M - 19.0.0-1775-g5e70124b 5f21c9264136 4460b22c7a56 2024-03-26T02:08:29.864 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi069.wpbewu smithi069 *:8443,9283,8765 running (10m) 86s ago 20m 455M - 19.0.0-1775-g5e70124b 5f21c9264136 380fd33842a8 2024-03-26T02:08:29.864 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi019 smithi019 running (10m) 30s ago 23m 65.8M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 c18934705774 2024-03-26T02:08:29.864 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi069 smithi069 running (9m) 86s ago 20m 48.1M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 a856bd263104 2024-03-26T02:08:29.864 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.0.0.smithi019.qbizxo smithi019 *:2049 running (2m) 30s ago 15m 70.8M - 5.7 5f21c9264136 601e946cdac2 2024-03-26T02:08:29.864 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.1.0.smithi069.uawdhc smithi069 *:2049 running (115s) 86s ago 15m 61.6M - 5.7 5f21c9264136 c8bc7245da23 2024-03-26T02:08:29.864 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi019 smithi019 *:9100 running (96s) 30s ago 21m 8228k - 1.5.0 0da6a335fe13 aac97ec99f0a 2024-03-26T02:08:29.865 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi069 smithi069 *:9100 running (90s) 86s ago 20m 3590k - 1.5.0 0da6a335fe13 6d0c5a6932ef 2024-03-26T02:08:29.865 INFO:teuthology.orchestra.run.smithi019.stdout:osd.0 smithi019 running (8m) 30s ago 20m 411M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 3f931cfe320a 2024-03-26T02:08:29.865 INFO:teuthology.orchestra.run.smithi019.stdout:osd.1 smithi019 running (7m) 30s ago 19m 375M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 758caf4ed841 2024-03-26T02:08:29.865 INFO:teuthology.orchestra.run.smithi019.stdout:osd.2 smithi019 running (6m) 30s ago 19m 384M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 6a9b701517ca 2024-03-26T02:08:29.865 INFO:teuthology.orchestra.run.smithi019.stdout:osd.3 smithi019 running (5m) 30s ago 18m 423M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 0401940d882b 2024-03-26T02:08:29.865 INFO:teuthology.orchestra.run.smithi019.stdout:osd.4 smithi069 running (4m) 86s ago 18m 344M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 6eb3b2ce9978 2024-03-26T02:08:29.865 INFO:teuthology.orchestra.run.smithi019.stdout:osd.5 smithi069 running (4m) 86s ago 17m 327M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 fb31f0e68fb7 2024-03-26T02:08:29.865 INFO:teuthology.orchestra.run.smithi019.stdout:osd.6 smithi069 running (3m) 86s ago 17m 302M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 924e2a35053c 2024-03-26T02:08:29.865 INFO:teuthology.orchestra.run.smithi019.stdout:osd.7 smithi069 running (3m) 86s ago 17m 287M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 d8052b0a18d6 2024-03-26T02:08:29.865 INFO:teuthology.orchestra.run.smithi019.stdout:prometheus.smithi019 smithi019 *:9095 running (71s) 30s ago 20m 46.4M - 2.43.0 a07b618ecd1d 86364cd8e9cf 2024-03-26T02:08:30.485 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:30 smithi019 ceph-mon[66359]: from='client.34626 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:08:30.507 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'ceph versions' 2024-03-26T02:08:30.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:30 smithi069 ceph-mon[53087]: from='client.34626 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:08:31.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:31 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:08:31.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:31 smithi019 ceph-mon[66359]: pgmap v398: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:31.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:31 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:08:31.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:31 smithi069 ceph-mon[53087]: pgmap v398: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:32.721 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:08:32.721 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": { 2024-03-26T02:08:32.721 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:08:32.721 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:08:32.721 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": { 2024-03-26T02:08:32.721 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:08:32.722 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:08:32.722 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": { 2024-03-26T02:08:32.722 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 8 2024-03-26T02:08:32.722 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:08:32.722 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": { 2024-03-26T02:08:32.722 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:08:32.722 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:08:32.722 INFO:teuthology.orchestra.run.smithi019.stdout: "overall": { 2024-03-26T02:08:32.722 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 14 2024-03-26T02:08:32.722 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-26T02:08:32.722 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:08:33.015 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:32 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/1526661311' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:33.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:32 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/1526661311' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:08:33.373 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-03-26T02:08:33.964 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:33 smithi019 ceph-mon[66359]: pgmap v399: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:34.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:33 smithi069 ceph-mon[53087]: pgmap v399: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:35.271 INFO:teuthology.orchestra.run.smithi019.stdout:wait for servicemap items w/ changing names to refresh 2024-03-26T02:08:35.271 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:35 smithi019 ceph-mon[66359]: pgmap v400: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:08:35.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:35 smithi069 ceph-mon[53087]: pgmap v400: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:08:35.880 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'sleep 60' 2024-03-26T02:08:37.298 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:37 smithi019 ceph-mon[66359]: pgmap v401: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:37.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:37 smithi069 ceph-mon[53087]: pgmap v401: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:39.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:39 smithi019 ceph-mon[66359]: pgmap v402: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:08:39.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:39 smithi069 ceph-mon[53087]: pgmap v402: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:08:41.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:41 smithi019 ceph-mon[66359]: pgmap v403: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:41.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:41 smithi069 ceph-mon[53087]: pgmap v403: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:43.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:43 smithi019 ceph-mon[66359]: pgmap v404: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:43.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:43 smithi069 ceph-mon[53087]: pgmap v404: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:45.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:45 smithi019 ceph-mon[66359]: pgmap v405: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:08:45.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:45 smithi069 ceph-mon[53087]: pgmap v405: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:08:46.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:46 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:08:46.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:46 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:08:47.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:47 smithi019 ceph-mon[66359]: pgmap v406: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:47.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:47 smithi069 ceph-mon[53087]: pgmap v406: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:49.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:49 smithi019 ceph-mon[66359]: pgmap v407: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:08:49.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:49 smithi069 ceph-mon[53087]: pgmap v407: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:08:51.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:51 smithi019 ceph-mon[66359]: pgmap v408: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:51.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:51 smithi069 ceph-mon[53087]: pgmap v408: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:53.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:53 smithi019 ceph-mon[66359]: pgmap v409: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:53.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:53 smithi069 ceph-mon[53087]: pgmap v409: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:55.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:55 smithi019 ceph-mon[66359]: pgmap v410: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:08:55.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:55 smithi069 ceph-mon[53087]: pgmap v410: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:08:57.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:57 smithi019 ceph-mon[66359]: pgmap v411: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:57.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:57 smithi069 ceph-mon[53087]: pgmap v411: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:08:59.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:08:59 smithi019 ceph-mon[66359]: pgmap v412: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:08:59.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:08:59 smithi069 ceph-mon[53087]: pgmap v412: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:09:01.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:00 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:09:01.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:00 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:09:02.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:01 smithi069 ceph-mon[53087]: pgmap v413: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:02.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:01 smithi019 ceph-mon[66359]: pgmap v413: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:03.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:03 smithi019 ceph-mon[66359]: pgmap v414: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:03.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:03 smithi069 ceph-mon[53087]: pgmap v414: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:05.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:05 smithi019 ceph-mon[66359]: pgmap v415: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:09:05.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:05 smithi069 ceph-mon[53087]: pgmap v415: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:09:06.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:06 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:09:06.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:06 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:06.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:06 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:09:06.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:06 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:09:06.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:06 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:09:06.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:06 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:09:06.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:06 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:06.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:06 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:09:06.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:06 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:09:06.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:06 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:09:07.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:07 smithi019 ceph-mon[66359]: pgmap v416: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:07.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:07 smithi069 ceph-mon[53087]: pgmap v416: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:09.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:09 smithi019 ceph-mon[66359]: pgmap v417: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:09:09.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:09 smithi069 ceph-mon[53087]: pgmap v417: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:09:11.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:11 smithi019 ceph-mon[66359]: pgmap v418: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:11.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:11 smithi069 ceph-mon[53087]: pgmap v418: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:13.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:13 smithi019 ceph-mon[66359]: pgmap v419: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:13.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:13 smithi069 ceph-mon[53087]: pgmap v419: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:15.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:15 smithi019 ceph-mon[66359]: pgmap v420: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:09:15.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:15 smithi069 ceph-mon[53087]: pgmap v420: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:09:16.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:16 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:09:16.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:16 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:09:17.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:17 smithi019 ceph-mon[66359]: pgmap v421: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:17.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:17 smithi069 ceph-mon[53087]: pgmap v421: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:19.517 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:19 smithi019 ceph-mon[66359]: pgmap v422: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:09:19.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:19 smithi069 ceph-mon[53087]: pgmap v422: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:09:21.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:21 smithi019 ceph-mon[66359]: pgmap v423: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:21.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:21 smithi069 ceph-mon[53087]: pgmap v423: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:23.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:23 smithi019 ceph-mon[66359]: pgmap v424: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:23.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:23 smithi069 ceph-mon[53087]: pgmap v424: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:25.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:25 smithi019 ceph-mon[66359]: pgmap v425: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:09:25.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:25 smithi069 ceph-mon[53087]: pgmap v425: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:09:27.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:27 smithi019 ceph-mon[66359]: pgmap v426: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:27.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:27 smithi069 ceph-mon[53087]: pgmap v426: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:29.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:29 smithi019 ceph-mon[66359]: pgmap v427: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:09:29.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:29 smithi069 ceph-mon[53087]: pgmap v427: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:09:31.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:30 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:09:31.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:30 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:09:32.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:31 smithi069 ceph-mon[53087]: pgmap v428: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:32.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:31 smithi019 ceph-mon[66359]: pgmap v428: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:33.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:33 smithi019 ceph-mon[66359]: pgmap v429: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:33.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:33 smithi069 ceph-mon[53087]: pgmap v429: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:35.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:35 smithi019 ceph-mon[66359]: pgmap v430: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:09:35.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:35 smithi069 ceph-mon[53087]: pgmap v430: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:09:37.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:37 smithi019 ceph-mon[66359]: pgmap v431: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:37.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:37 smithi069 ceph-mon[53087]: pgmap v431: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:38.375 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'ceph orch ps' 2024-03-26T02:09:39.367 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:39 smithi019 ceph-mon[66359]: pgmap v432: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:09:39.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:39 smithi069 ceph-mon[53087]: pgmap v432: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:09:40.423 INFO:teuthology.orchestra.run.smithi019.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:09:40.423 INFO:teuthology.orchestra.run.smithi019.stdout:alertmanager.smithi019 smithi019 *:9093,9094 running (2m) 100s ago 22m 15.2M - 0.25.0 c8568f914cd2 2cb7fa4e7f1c 2024-03-26T02:09:40.423 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi019 smithi019 running (9m) 100s ago 22m 6660k - 19.0.0-1775-g5e70124b 5f21c9264136 c5f91a47ddf2 2024-03-26T02:09:40.423 INFO:teuthology.orchestra.run.smithi019.stdout:crash.smithi069 smithi069 running (9m) 2m ago 22m 6669k - 19.0.0-1775-g5e70124b 5f21c9264136 05128803b49e 2024-03-26T02:09:40.423 INFO:teuthology.orchestra.run.smithi019.stdout:grafana.smithi019 smithi019 *:3000 running (106s) 100s ago 22m 51.5M - 9.4.12 a0ea5e3c46d9 cdb65c966500 2024-03-26T02:09:40.423 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi019.lmryom smithi019 running (3m) 100s ago 16m 36.7M - 19.0.0-1775-g5e70124b 5f21c9264136 499227af5537 2024-03-26T02:09:40.423 INFO:teuthology.orchestra.run.smithi019.stdout:mds.foofs.smithi069.spgwgo smithi069 running (3m) 2m ago 16m 11.5M - 19.0.0-1775-g5e70124b 5f21c9264136 b8adfbbfa78e 2024-03-26T02:09:40.423 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi019.ixuvdm smithi019 *:8443,9283,8765 running (12m) 100s ago 24m 517M - 19.0.0-1775-g5e70124b 5f21c9264136 4460b22c7a56 2024-03-26T02:09:40.424 INFO:teuthology.orchestra.run.smithi019.stdout:mgr.smithi069.wpbewu smithi069 *:8443,9283,8765 running (11m) 2m ago 22m 455M - 19.0.0-1775-g5e70124b 5f21c9264136 380fd33842a8 2024-03-26T02:09:40.424 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi019 smithi019 running (11m) 100s ago 24m 65.8M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 c18934705774 2024-03-26T02:09:40.424 INFO:teuthology.orchestra.run.smithi019.stdout:mon.smithi069 smithi069 running (10m) 2m ago 22m 48.1M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 a856bd263104 2024-03-26T02:09:40.424 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.0.0.smithi019.qbizxo smithi019 *:2049 running (3m) 100s ago 16m 70.8M - 5.7 5f21c9264136 601e946cdac2 2024-03-26T02:09:40.424 INFO:teuthology.orchestra.run.smithi019.stdout:nfs.foo.1.0.smithi069.uawdhc smithi069 *:2049 running (3m) 2m ago 16m 61.6M - 5.7 5f21c9264136 c8bc7245da23 2024-03-26T02:09:40.424 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi019 smithi019 *:9100 running (2m) 100s ago 22m 8228k - 1.5.0 0da6a335fe13 aac97ec99f0a 2024-03-26T02:09:40.424 INFO:teuthology.orchestra.run.smithi019.stdout:node-exporter.smithi069 smithi069 *:9100 running (2m) 2m ago 21m 3590k - 1.5.0 0da6a335fe13 6d0c5a6932ef 2024-03-26T02:09:40.424 INFO:teuthology.orchestra.run.smithi019.stdout:osd.0 smithi019 running (9m) 100s ago 21m 411M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 3f931cfe320a 2024-03-26T02:09:40.424 INFO:teuthology.orchestra.run.smithi019.stdout:osd.1 smithi019 running (8m) 100s ago 20m 375M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 758caf4ed841 2024-03-26T02:09:40.424 INFO:teuthology.orchestra.run.smithi019.stdout:osd.2 smithi019 running (7m) 100s ago 20m 384M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 6a9b701517ca 2024-03-26T02:09:40.424 INFO:teuthology.orchestra.run.smithi019.stdout:osd.3 smithi019 running (6m) 100s ago 19m 423M 1937M 19.0.0-1775-g5e70124b 5f21c9264136 0401940d882b 2024-03-26T02:09:40.424 INFO:teuthology.orchestra.run.smithi019.stdout:osd.4 smithi069 running (6m) 2m ago 19m 344M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 6eb3b2ce9978 2024-03-26T02:09:40.424 INFO:teuthology.orchestra.run.smithi019.stdout:osd.5 smithi069 running (5m) 2m ago 19m 327M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 fb31f0e68fb7 2024-03-26T02:09:40.424 INFO:teuthology.orchestra.run.smithi019.stdout:osd.6 smithi069 running (4m) 2m ago 18m 302M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 924e2a35053c 2024-03-26T02:09:40.425 INFO:teuthology.orchestra.run.smithi019.stdout:osd.7 smithi069 running (4m) 2m ago 18m 287M 2704M 19.0.0-1775-g5e70124b 5f21c9264136 d8052b0a18d6 2024-03-26T02:09:40.425 INFO:teuthology.orchestra.run.smithi019.stdout:prometheus.smithi019 smithi019 *:9095 running (2m) 100s ago 21m 46.4M - 2.43.0 a07b618ecd1d 86364cd8e9cf 2024-03-26T02:09:41.099 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'ceph orch upgrade status' 2024-03-26T02:09:41.414 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:41 smithi019 ceph-mon[66359]: from='client.34634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:09:41.415 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:41 smithi019 ceph-mon[66359]: pgmap v433: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:41.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:41 smithi069 ceph-mon[53087]: from='client.34634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:09:41.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:41 smithi069 ceph-mon[53087]: pgmap v433: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:43.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:43 smithi019 ceph-mon[66359]: pgmap v434: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:43.302 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:09:43.302 INFO:teuthology.orchestra.run.smithi019.stdout: "target_image": null, 2024-03-26T02:09:43.302 INFO:teuthology.orchestra.run.smithi019.stdout: "in_progress": false, 2024-03-26T02:09:43.302 INFO:teuthology.orchestra.run.smithi019.stdout: "which": "", 2024-03-26T02:09:43.302 INFO:teuthology.orchestra.run.smithi019.stdout: "services_complete": [], 2024-03-26T02:09:43.302 INFO:teuthology.orchestra.run.smithi019.stdout: "progress": null, 2024-03-26T02:09:43.302 INFO:teuthology.orchestra.run.smithi019.stdout: "message": "", 2024-03-26T02:09:43.302 INFO:teuthology.orchestra.run.smithi019.stdout: "is_paused": false 2024-03-26T02:09:43.302 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:09:43.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:43 smithi069 ceph-mon[53087]: pgmap v434: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:44.004 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'ceph health detail' 2024-03-26T02:09:44.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:44 smithi019 ceph-mon[66359]: from='client.34638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:09:44.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:44 smithi069 ceph-mon[53087]: from='client.34638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:09:45.305 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:45 smithi019 ceph-mon[66359]: pgmap v435: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:09:45.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:45 smithi069 ceph-mon[53087]: pgmap v435: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:09:46.182 INFO:teuthology.orchestra.run.smithi019.stdout:HEALTH_OK 2024-03-26T02:09:46.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:46 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:09:46.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:46 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/3371331039' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:09:46.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:46 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:09:46.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:46 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/3371331039' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:09:46.847 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'ceph versions' 2024-03-26T02:09:47.416 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:47 smithi019 ceph-mon[66359]: pgmap v436: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:47.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:47 smithi069 ceph-mon[53087]: pgmap v436: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:48.942 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-03-26T02:09:48.943 INFO:teuthology.orchestra.run.smithi019.stdout: "mon": { 2024-03-26T02:09:48.943 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:09:48.943 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:09:48.943 INFO:teuthology.orchestra.run.smithi019.stdout: "mgr": { 2024-03-26T02:09:48.943 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:09:48.943 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:09:48.943 INFO:teuthology.orchestra.run.smithi019.stdout: "osd": { 2024-03-26T02:09:48.943 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 8 2024-03-26T02:09:48.943 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:09:48.943 INFO:teuthology.orchestra.run.smithi019.stdout: "mds": { 2024-03-26T02:09:48.943 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T02:09:48.943 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:09:48.943 INFO:teuthology.orchestra.run.smithi019.stdout: "overall": { 2024-03-26T02:09:48.944 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 14 2024-03-26T02:09:48.944 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-26T02:09:48.944 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-03-26T02:09:49.251 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:48 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/1872754045' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:09:49.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:48 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/1872754045' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:09:49.608 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-03-26T02:09:50.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:49 smithi019 ceph-mon[66359]: pgmap v437: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:09:50.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:49 smithi069 ceph-mon[53087]: pgmap v437: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:09:51.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:51 smithi019 ceph-mon[66359]: pgmap v438: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:51.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:51 smithi069 ceph-mon[53087]: pgmap v438: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:51.722 INFO:teuthology.orchestra.run.smithi019.stdout:true 2024-03-26T02:09:52.358 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:52 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/1686820595' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:09:52.381 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-03-26T02:09:52.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:52 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/1686820595' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:09:53.356 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:53 smithi019 ceph-mon[66359]: pgmap v439: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:53.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:53 smithi069 ceph-mon[53087]: pgmap v439: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:54.441 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)" 2024-03-26T02:09:54.731 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:54 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/345851962' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:09:54.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:54 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/345851962' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:09:55.059 INFO:teuthology.task.sequential:In sequential, running task cephadm.wait_for_service... 2024-03-26T02:09:55.060 INFO:tasks.cephadm:Waiting for ceph service nfs.foo to start (timeout 300)... 2024-03-26T02:09:55.061 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- ceph orch ls -f json 2024-03-26T02:09:55.692 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:55 smithi019 ceph-mon[66359]: pgmap v440: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:09:55.859 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:55 smithi069 ceph-mon[53087]: pgmap v440: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:09:57.128 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-26T02:09:57.128 INFO:teuthology.orchestra.run.smithi019.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-03-26T01:46:13.250596Z", "last_refresh": "2024-03-26T02:07:59.740258Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-03-26T01:46:06.931254Z", "last_refresh": "2024-03-26T02:07:03.108772Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "spec": {"anonymous_access": true, "protocol": "https"}, "status": {"created": "2024-03-26T01:46:09.898597Z", "last_refresh": "2024-03-26T02:07:59.740420Z", "ports": [3000], "running": 1, "size": 1}}, {"placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-03-26T01:52:48.827688Z", "last_refresh": "2024-03-26T02:07:03.109462Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-03-26T01:46:05.586775Z", "last_refresh": "2024-03-26T02:07:03.108896Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi019:172.21.15.19=smithi019", "smithi069:172.21.15.69=smithi069"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-03-26T01:46:56.697362Z", "last_refresh": "2024-03-26T02:07:03.108983Z", "running": 2, "size": 2}}, {"events": ["2024-03-26T02:09:06.102686Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 2049}, "status": {"created": "2024-03-26T01:53:09.800546Z", "last_refresh": "2024-03-26T02:07:03.109541Z", "ports": [2049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-03-26T01:46:11.134728Z", "last_refresh": "2024-03-26T02:07:03.109064Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767", "container_image_name": "quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2", "last_refresh": "2024-03-26T02:07:03.109146Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-03-26T01:46:08.313914Z", "last_refresh": "2024-03-26T02:07:59.740575Z", "ports": [9095], "running": 1, "size": 1}}] 2024-03-26T02:09:57.445 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:57 smithi019 ceph-mon[66359]: pgmap v441: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:57.445 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:57 smithi019 ceph-mon[66359]: from='client.34658 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T02:09:57.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:57 smithi069 ceph-mon[53087]: pgmap v441: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:09:57.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:57 smithi069 ceph-mon[53087]: from='client.34658 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T02:09:57.767 INFO:tasks.cephadm:nfs.foo has 2/2 2024-03-26T02:09:57.768 DEBUG:teuthology.parallel:result is None 2024-03-26T02:09:57.768 INFO:teuthology.run_tasks:Running task vip.exec... 2024-03-26T02:09:57.779 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi019.front.sepia.ceph.com 2024-03-26T02:09:57.779 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'umount /mnt/foo' 2024-03-26T02:09:57.807 INFO:teuthology.orchestra.run.smithi019.stderr:+ umount /mnt/foo 2024-03-26T02:09:57.818 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-26T02:09:57.827 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi019.front.sepia.ceph.com 2024-03-26T02:09:57.828 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- bash -c 'ceph nfs cluster ls | grep foo' 2024-03-26T02:09:59.430 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:09:59 smithi019 ceph-mon[66359]: pgmap v442: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:09:59.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:09:59 smithi069 ceph-mon[53087]: pgmap v442: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:09:59.887 INFO:teuthology.orchestra.run.smithi019.stdout: "foo" 2024-03-26T02:10:00.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:10:00 smithi019 ceph-mon[66359]: from='client.34662 -' entity='client.admin' cmd=[{"prefix": "nfs cluster ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:10:00.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:10:00 smithi019 ceph-mon[66359]: overall HEALTH_OK 2024-03-26T02:10:00.567 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- bash -c 'ceph nfs export ls foo --detailed' 2024-03-26T02:10:00.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:10:00 smithi069 ceph-mon[53087]: from='client.34662 -' entity='client.admin' cmd=[{"prefix": "nfs cluster ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:10:00.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:10:00 smithi069 ceph-mon[53087]: overall HEALTH_OK 2024-03-26T02:10:01.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:10:01 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:10:01.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:10:01 smithi019 ceph-mon[66359]: pgmap v443: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:10:01.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:10:01 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:10:01.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:10:01 smithi069 ceph-mon[53087]: pgmap v443: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:10:02.747 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-03-26T02:10:02.747 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-03-26T02:10:02.747 INFO:teuthology.orchestra.run.smithi019.stdout: "access_type": "RW", 2024-03-26T02:10:02.747 INFO:teuthology.orchestra.run.smithi019.stdout: "clients": [], 2024-03-26T02:10:02.747 INFO:teuthology.orchestra.run.smithi019.stdout: "cluster_id": "foo", 2024-03-26T02:10:02.747 INFO:teuthology.orchestra.run.smithi019.stdout: "export_id": 1, 2024-03-26T02:10:02.748 INFO:teuthology.orchestra.run.smithi019.stdout: "fsal": { 2024-03-26T02:10:02.748 INFO:teuthology.orchestra.run.smithi019.stdout: "fs_name": "foofs", 2024-03-26T02:10:02.748 INFO:teuthology.orchestra.run.smithi019.stdout: "name": "CEPH", 2024-03-26T02:10:02.748 INFO:teuthology.orchestra.run.smithi019.stdout: "user_id": "nfs.foo.1" 2024-03-26T02:10:02.748 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-03-26T02:10:02.748 INFO:teuthology.orchestra.run.smithi019.stdout: "path": "/", 2024-03-26T02:10:02.748 INFO:teuthology.orchestra.run.smithi019.stdout: "protocols": [ 2024-03-26T02:10:02.748 INFO:teuthology.orchestra.run.smithi019.stdout: 4 2024-03-26T02:10:02.748 INFO:teuthology.orchestra.run.smithi019.stdout: ], 2024-03-26T02:10:02.748 INFO:teuthology.orchestra.run.smithi019.stdout: "pseudo": "/fake", 2024-03-26T02:10:02.749 INFO:teuthology.orchestra.run.smithi019.stdout: "security_label": true, 2024-03-26T02:10:02.749 INFO:teuthology.orchestra.run.smithi019.stdout: "squash": "none", 2024-03-26T02:10:02.749 INFO:teuthology.orchestra.run.smithi019.stdout: "transports": [ 2024-03-26T02:10:02.749 INFO:teuthology.orchestra.run.smithi019.stdout: "TCP" 2024-03-26T02:10:02.749 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-03-26T02:10:02.749 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-03-26T02:10:02.749 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-03-26T02:10:03.383 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:10:03 smithi019 ceph-mon[66359]: from='client.34666 -' entity='client.admin' cmd=[{"prefix": "nfs export ls", "cluster_id": "foo", "detailed": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:10:03.383 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:10:03 smithi019 ceph-mon[66359]: pgmap v444: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:10:03.406 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- bash -c 'rados -p .nfs --all ls -' 2024-03-26T02:10:03.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:10:03 smithi069 ceph-mon[53087]: from='client.34666 -' entity='client.admin' cmd=[{"prefix": "nfs export ls", "cluster_id": "foo", "detailed": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:10:03.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:10:03 smithi069 ceph-mon[53087]: pgmap v444: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-26T02:10:05.169 INFO:teuthology.orchestra.run.smithi019.stdout:foo rec-0000000000000004:nfs.foo.1 2024-03-26T02:10:05.169 INFO:teuthology.orchestra.run.smithi019.stdout:foo grace 2024-03-26T02:10:05.169 INFO:teuthology.orchestra.run.smithi019.stdout:foo rec-0000000000000004:nfs.foo.0 2024-03-26T02:10:05.169 INFO:teuthology.orchestra.run.smithi019.stdout:foo conf-nfs.foo 2024-03-26T02:10:05.169 INFO:teuthology.orchestra.run.smithi019.stdout:foo export-1 2024-03-26T02:10:05.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:10:05 smithi019 ceph-mon[66359]: mgrmap e42: smithi019.ixuvdm(active, since 11m), standbys: smithi069.wpbewu 2024-03-26T02:10:05.265 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:10:05 smithi019 ceph-mon[66359]: pgmap v445: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:10:05.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:10:05 smithi069 ceph-mon[53087]: mgrmap e42: smithi019.ixuvdm(active, since 11m), standbys: smithi069.wpbewu 2024-03-26T02:10:05.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:10:05 smithi069 ceph-mon[53087]: pgmap v445: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:10:05.909 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-26T02:10:05.919 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi019.front.sepia.ceph.com 2024-03-26T02:10:05.920 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 692607c6-eb12-11ee-95d0-87774f69a715 -- bash -c 'set -ex 2024-03-26T02:10:05.920 DEBUG:teuthology.orchestra.run.smithi019:> [[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]] 2024-03-26T02:10:05.920 DEBUG:teuthology.orchestra.run.smithi019:> ' 2024-03-26T02:10:06.459 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:10:06 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:10:06.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:10:06 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:10:07.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:10:07 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:10:07.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:10:07 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:10:07.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:10:07 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:10:07.515 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:10:07 smithi019 ceph-mon[66359]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:10:07.516 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:10:07 smithi019 ceph-mon[66359]: pgmap v446: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:10:07.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:10:07 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:10:07.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:10:07 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:10:07.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:10:07 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:10:07.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:10:07 smithi069 ceph-mon[53087]: from='mgr.34104 172.21.15.19:0/1113559180' entity='mgr.smithi019.ixuvdm' 2024-03-26T02:10:07.609 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:10:07 smithi069 ceph-mon[53087]: pgmap v446: 97 pgs: 97 active+clean; 214 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:10:07.734 INFO:teuthology.orchestra.run.smithi019.stderr:++ ceph config get mgr mgr/cephadm/migration_current 2024-03-26T02:10:08.014 INFO:teuthology.orchestra.run.smithi019.stderr:+ [[ 6 -gt 2 ]] 2024-03-26T02:10:08.669 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-26T02:10:08.679 INFO:tasks.cephadm:Teardown begin 2024-03-26T02:10:08.680 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T02:10:08.708 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T02:10:08.743 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-26T02:10:08.744 DEBUG:teuthology.orchestra.run.smithi019:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T02:10:08.763 DEBUG:teuthology.orchestra.run.smithi069:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T02:10:08.800 INFO:tasks.cephadm:Stopping all daemons... 2024-03-26T02:10:08.801 INFO:tasks.cephadm.mon.smithi019:Stopping mon.smithi019... 2024-03-26T02:10:08.801 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ceph-692607c6-eb12-11ee-95d0-87774f69a715@mon.smithi019 2024-03-26T02:10:08.951 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:10:08 smithi019 systemd[1]: Stopping Ceph mon.smithi019 for 692607c6-eb12-11ee-95d0-87774f69a715... 2024-03-26T02:10:09.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:10:08 smithi069 ceph-mon[53087]: from='client.? 172.21.15.19:0/2079442970' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/cephadm/migration_current"}]: dispatch 2024-03-26T02:10:09.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:10:08 smithi019 ceph-mon[66359]: from='client.? 172.21.15.19:0/2079442970' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/cephadm/migration_current"}]: dispatch 2024-03-26T02:10:09.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:10:09 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi019[66355]: 2024-03-26T02:10:09.148+0000 7fa5e89fe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi019 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:10:09.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:10:09 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi019[66355]: 2024-03-26T02:10:09.148+0000 7fa5e89fe700 -1 mon.smithi019@0(leader) e3 *** Got Signal Terminated *** 2024-03-26T02:10:09.266 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:10:09 smithi019 podman[111370]: 2024-03-26 02:10:09.169835282 +0000 UTC m=+0.142229108 container died c18934705774976b47d22d796fcba2731576c23adc1015d6071f9e880e27e628 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi019, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.build-date=20240319) 2024-03-26T02:10:09.826 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:10:09 smithi019 podman[111370]: 2024-03-26 02:10:09.566234802 +0000 UTC m=+0.538628629 container cleanup c18934705774976b47d22d796fcba2731576c23adc1015d6071f9e880e27e628 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi019, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-5e70124) 2024-03-26T02:10:09.826 INFO:journalctl@ceph.mon.smithi019.smithi019.stdout:Mar 26 02:10:09 smithi019 bash[111370]: ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi019 2024-03-26T02:10:10.014 DEBUG:teuthology.orchestra.run.smithi019:> sudo pkill -f 'journalctl -f -n 0 -u ceph-692607c6-eb12-11ee-95d0-87774f69a715@mon.smithi019.service' 2024-03-26T02:10:10.062 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:10:10.063 INFO:tasks.cephadm.mon.smithi019:Stopped mon.smithi019 2024-03-26T02:10:10.063 INFO:tasks.cephadm.mon.smithi069:Stopping mon.smithi069... 2024-03-26T02:10:10.063 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-692607c6-eb12-11ee-95d0-87774f69a715@mon.smithi069 2024-03-26T02:10:10.359 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:10:10 smithi069 systemd[1]: Stopping Ceph mon.smithi069 for 692607c6-eb12-11ee-95d0-87774f69a715... 2024-03-26T02:10:10.833 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:10:10 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi069[53083]: 2024-03-26T02:10:10.522+0000 7fbcd584b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi069 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:10:10.833 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:10:10 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi069[53083]: 2024-03-26T02:10:10.522+0000 7fbcd584b700 -1 mon.smithi069@1(peon) e3 *** Got Signal Terminated *** 2024-03-26T02:10:10.833 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:10:10 smithi069 podman[79352]: 2024-03-26 02:10:10.57435221 +0000 UTC m=+0.315485958 container died a856bd263104279c177520817cefc196208cf47d085ab195d3cd798b608d2301 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi069, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319) 2024-03-26T02:10:11.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:10:11 smithi069 podman[79352]: 2024-03-26 02:10:11.006765848 +0000 UTC m=+0.747899602 container cleanup a856bd263104279c177520817cefc196208cf47d085ab195d3cd798b608d2301 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi069, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-26T02:10:11.109 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:10:11 smithi069 bash[79352]: ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi069 2024-03-26T02:10:11.442 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-692607c6-eb12-11ee-95d0-87774f69a715@mon.smithi069.service' 2024-03-26T02:10:11.466 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:10:11 smithi069 podman[79366]: 2024-03-26 02:10:11.208295542 +0000 UTC m=+0.638531570 container remove a856bd263104279c177520817cefc196208cf47d085ab195d3cd798b608d2301 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-mon-smithi069, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5) 2024-03-26T02:10:11.467 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:10:11 smithi069 systemd[1]: ceph-692607c6-eb12-11ee-95d0-87774f69a715@mon.smithi069.service: Deactivated successfully. 2024-03-26T02:10:11.467 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:10:11 smithi069 systemd[1]: Stopped Ceph mon.smithi069 for 692607c6-eb12-11ee-95d0-87774f69a715. 2024-03-26T02:10:11.467 INFO:journalctl@ceph.mon.smithi069.smithi069.stdout:Mar 26 02:10:11 smithi069 systemd[1]: ceph-692607c6-eb12-11ee-95d0-87774f69a715@mon.smithi069.service: Consumed 6.221s CPU time. 2024-03-26T02:10:11.492 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:10:11.492 INFO:tasks.cephadm.mon.smithi069:Stopped mon.smithi069 2024-03-26T02:10:11.492 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-26T02:10:11.492 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.0 2024-03-26T02:10:11.765 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:10:11 smithi019 systemd[1]: Stopping Ceph osd.0 for 692607c6-eb12-11ee-95d0-87774f69a715... 2024-03-26T02:10:12.265 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:10:11 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0[72511]: 2024-03-26T02:10:11.829+0000 7fb5640d8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:10:12.265 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:10:11 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0[72511]: 2024-03-26T02:10:11.829+0000 7fb5640d8700 -1 osd.0 168 *** Got signal Terminated *** 2024-03-26T02:10:12.266 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:10:11 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0[72511]: 2024-03-26T02:10:11.829+0000 7fb5640d8700 -1 osd.0 168 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:10:17.597 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:10:17 smithi019 podman[111510]: 2024-03-26 02:10:17.314268189 +0000 UTC m=+5.625165966 container died 3f931cfe320ac0caf9266b9378644dbdc715b0d5494045f3906ee69686a30a07 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-26T02:10:17.981 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:10:17 smithi019 podman[111510]: 2024-03-26 02:10:17.687666863 +0000 UTC m=+5.998564634 container cleanup 3f931cfe320ac0caf9266b9378644dbdc715b0d5494045f3906ee69686a30a07 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-26T02:10:17.981 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:10:17 smithi019 bash[111510]: ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0 2024-03-26T02:10:18.265 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:10:17 smithi019 podman[111530]: 2024-03-26 02:10:17.98115875 +0000 UTC m=+0.668849849 container remove 3f931cfe320ac0caf9266b9378644dbdc715b0d5494045f3906ee69686a30a07 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-26T02:10:18.634 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:10:18 smithi019 podman[111618]: 2024-03-26 02:10:18.212237163 +0000 UTC m=+0.018479721 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:10:18.635 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:10:18 smithi019 podman[111618]: 2024-03-26 02:10:18.332117121 +0000 UTC m=+0.138359679 container create e013ad55c867239f052db36539573da279c058c6959cdb8400d60181a37012d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0-deactivate, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:10:18.635 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:10:18 smithi019 podman[111618]: 2024-03-26 02:10:18.5516195 +0000 UTC m=+0.357862072 container init e013ad55c867239f052db36539573da279c058c6959cdb8400d60181a37012d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-26T02:10:18.635 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:10:18 smithi019 podman[111618]: 2024-03-26 02:10:18.555176836 +0000 UTC m=+0.361419398 container start e013ad55c867239f052db36539573da279c058c6959cdb8400d60181a37012d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:10:18.635 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:10:18 smithi019 podman[111618]: 2024-03-26 02:10:18.634552253 +0000 UTC m=+0.440794828 container attach e013ad55c867239f052db36539573da279c058c6959cdb8400d60181a37012d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True) 2024-03-26T02:10:19.015 INFO:journalctl@ceph.osd.0.smithi019.stdout:Mar 26 02:10:18 smithi019 podman[111618]: 2024-03-26 02:10:18.856038519 +0000 UTC m=+0.662281082 container died e013ad55c867239f052db36539573da279c058c6959cdb8400d60181a37012d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-26T02:10:19.481 DEBUG:teuthology.orchestra.run.smithi019:> sudo pkill -f 'journalctl -f -n 0 -u ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.0.service' 2024-03-26T02:10:19.523 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:10:19.524 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-26T02:10:19.524 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-26T02:10:19.524 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.1 2024-03-26T02:10:19.765 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:10:19 smithi019 systemd[1]: Stopping Ceph osd.1 for 692607c6-eb12-11ee-95d0-87774f69a715... 2024-03-26T02:10:20.265 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:10:19 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1[75084]: 2024-03-26T02:10:19.873+0000 7f85cf6fa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:10:20.265 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:10:19 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1[75084]: 2024-03-26T02:10:19.873+0000 7f85cf6fa700 -1 osd.1 168 *** Got signal Terminated *** 2024-03-26T02:10:20.266 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:10:19 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1[75084]: 2024-03-26T02:10:19.873+0000 7f85cf6fa700 -1 osd.1 168 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:10:25.743 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:10:25 smithi019 podman[111724]: 2024-03-26 02:10:25.462910917 +0000 UTC m=+5.709439817 container died 758caf4ed841dfa52bd3a8115b7b0108fc27e1ad2ae2beadd60247c45dc57d9e (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:10:26.015 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:10:25 smithi019 podman[111724]: 2024-03-26 02:10:25.874446863 +0000 UTC m=+6.120975753 container cleanup 758caf4ed841dfa52bd3a8115b7b0108fc27e1ad2ae2beadd60247c45dc57d9e (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-26T02:10:26.015 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:10:25 smithi019 bash[111724]: ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1 2024-03-26T02:10:26.477 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:10:26 smithi019 podman[111743]: 2024-03-26 02:10:26.109195905 +0000 UTC m=+0.649441683 container remove 758caf4ed841dfa52bd3a8115b7b0108fc27e1ad2ae2beadd60247c45dc57d9e (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-26T02:10:26.729 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:10:26 smithi019 podman[111834]: 2024-03-26 02:10:26.382591557 +0000 UTC m=+0.020911955 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:10:26.729 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:10:26 smithi019 podman[111834]: 2024-03-26 02:10:26.510068718 +0000 UTC m=+0.148389103 container create f4e7b9a40aa7b9cd975a98d4a40d23bdfb25c747167ae7cbaeabbab01aa07b34 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-26T02:10:26.730 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:10:26 smithi019 podman[111834]: 2024-03-26 02:10:26.72952169 +0000 UTC m=+0.367842072 container init f4e7b9a40aa7b9cd975a98d4a40d23bdfb25c747167ae7cbaeabbab01aa07b34 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:10:27.016 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:10:26 smithi019 podman[111834]: 2024-03-26 02:10:26.732749274 +0000 UTC m=+0.371069653 container start f4e7b9a40aa7b9cd975a98d4a40d23bdfb25c747167ae7cbaeabbab01aa07b34 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2024-03-26T02:10:27.016 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:10:26 smithi019 podman[111834]: 2024-03-26 02:10:26.821256252 +0000 UTC m=+0.459576640 container attach f4e7b9a40aa7b9cd975a98d4a40d23bdfb25c747167ae7cbaeabbab01aa07b34 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:10:27.333 INFO:journalctl@ceph.osd.1.smithi019.stdout:Mar 26 02:10:27 smithi019 podman[111834]: 2024-03-26 02:10:27.034066994 +0000 UTC m=+0.672387371 container died f4e7b9a40aa7b9cd975a98d4a40d23bdfb25c747167ae7cbaeabbab01aa07b34 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:10:27.624 DEBUG:teuthology.orchestra.run.smithi019:> sudo pkill -f 'journalctl -f -n 0 -u ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.1.service' 2024-03-26T02:10:27.663 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:10:27.663 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-26T02:10:27.664 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-26T02:10:27.664 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.2 2024-03-26T02:10:28.003 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:10:27 smithi019 systemd[1]: Stopping Ceph osd.2 for 692607c6-eb12-11ee-95d0-87774f69a715... 2024-03-26T02:10:28.265 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:10:28 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2[77755]: 2024-03-26T02:10:28.000+0000 7f6101a47700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:10:28.265 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:10:28 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2[77755]: 2024-03-26T02:10:28.000+0000 7f6101a47700 -1 osd.2 168 *** Got signal Terminated *** 2024-03-26T02:10:28.265 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:10:28 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2[77755]: 2024-03-26T02:10:28.000+0000 7f6101a47700 -1 osd.2 168 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:10:33.778 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:10:33 smithi019 podman[111972]: 2024-03-26 02:10:33.521564589 +0000 UTC m=+5.623821922 container died 6a9b701517ca8a8d5a0da2a333a2594a2dc289baf7918e273b577e3191576471 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-26T02:10:34.095 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:10:33 smithi019 podman[111972]: 2024-03-26 02:10:33.868766115 +0000 UTC m=+5.971023434 container cleanup 6a9b701517ca8a8d5a0da2a333a2594a2dc289baf7918e273b577e3191576471 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-26T02:10:34.095 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:10:33 smithi019 bash[111972]: ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2 2024-03-26T02:10:34.515 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:10:34 smithi019 podman[111990]: 2024-03-26 02:10:34.095076919 +0000 UTC m=+0.568989007 container remove 6a9b701517ca8a8d5a0da2a333a2594a2dc289baf7918e273b577e3191576471 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-26T02:10:34.866 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:10:34 smithi019 podman[112078]: 2024-03-26 02:10:34.458783949 +0000 UTC m=+0.018613936 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:10:34.866 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:10:34 smithi019 podman[112078]: 2024-03-26 02:10:34.612934339 +0000 UTC m=+0.172764332 container create 86149430e0814cd7bac85502194fb907aaed79e9076b9f678b90a3bcaad29660 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=squid-5e70124) 2024-03-26T02:10:34.867 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:10:34 smithi019 podman[112078]: 2024-03-26 02:10:34.865498255 +0000 UTC m=+0.425328231 container init 86149430e0814cd7bac85502194fb907aaed79e9076b9f678b90a3bcaad29660 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, ceph=True) 2024-03-26T02:10:35.180 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:10:34 smithi019 podman[112078]: 2024-03-26 02:10:34.868962821 +0000 UTC m=+0.428792800 container start 86149430e0814cd7bac85502194fb907aaed79e9076b9f678b90a3bcaad29660 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2-deactivate, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-26T02:10:35.180 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:10:34 smithi019 podman[112078]: 2024-03-26 02:10:34.948833918 +0000 UTC m=+0.508663905 container attach 86149430e0814cd7bac85502194fb907aaed79e9076b9f678b90a3bcaad29660 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-26T02:10:35.477 INFO:journalctl@ceph.osd.2.smithi019.stdout:Mar 26 02:10:35 smithi019 podman[112078]: 2024-03-26 02:10:35.18056134 +0000 UTC m=+0.740391319 container died 86149430e0814cd7bac85502194fb907aaed79e9076b9f678b90a3bcaad29660 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:10:35.762 DEBUG:teuthology.orchestra.run.smithi019:> sudo pkill -f 'journalctl -f -n 0 -u ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.2.service' 2024-03-26T02:10:35.801 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:10:35.802 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-26T02:10:35.802 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-26T02:10:35.802 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.3 2024-03-26T02:10:36.015 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:10:35 smithi019 systemd[1]: Stopping Ceph osd.3 for 692607c6-eb12-11ee-95d0-87774f69a715... 2024-03-26T02:10:36.515 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:10:36 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3[80354]: 2024-03-26T02:10:36.146+0000 7f3b85378700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:10:36.515 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:10:36 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3[80354]: 2024-03-26T02:10:36.146+0000 7f3b85378700 -1 osd.3 168 *** Got signal Terminated *** 2024-03-26T02:10:36.515 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:10:36 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3[80354]: 2024-03-26T02:10:36.146+0000 7f3b85378700 -1 osd.3 168 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:10:38.859 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:38 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:38.558+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:14.972652+0000 front 2024-03-26T02:10:14.972635+0000 (oldest deadline 2024-03-26T02:10:37.872709+0000) 2024-03-26T02:10:39.515 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:39 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:39.253+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:10:39.859 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:39 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:39.514+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:14.972652+0000 front 2024-03-26T02:10:14.972635+0000 (oldest deadline 2024-03-26T02:10:37.872709+0000) 2024-03-26T02:10:40.219 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:40 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:40.070+0000 7fdda3bf5700 -1 osd.5 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.990521+0000 front 2024-03-26T02:10:15.990504+0000 (oldest deadline 2024-03-26T02:10:39.490509+0000) 2024-03-26T02:10:40.265 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:10:39 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3[80354]: 2024-03-26T02:10:39.783+0000 7f3b8196c700 -1 osd.3 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:13.002254+0000 front 2024-03-26T02:10:13.002237+0000 (oldest deadline 2024-03-26T02:10:38.902371+0000) 2024-03-26T02:10:40.517 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:40 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:40.218+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:10:40.859 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:40 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:40.516+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:14.972652+0000 front 2024-03-26T02:10:14.972635+0000 (oldest deadline 2024-03-26T02:10:37.872709+0000) 2024-03-26T02:10:41.015 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:10:40 smithi019 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3[80354]: 2024-03-26T02:10:40.760+0000 7f3b8196c700 -1 osd.3 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:13.002254+0000 front 2024-03-26T02:10:13.002237+0000 (oldest deadline 2024-03-26T02:10:38.902371+0000) 2024-03-26T02:10:41.359 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:41 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:41.077+0000 7fdda3bf5700 -1 osd.5 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.990521+0000 front 2024-03-26T02:10:15.990504+0000 (oldest deadline 2024-03-26T02:10:39.490509+0000) 2024-03-26T02:10:41.360 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:41 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:41.258+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:10:41.768 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:10:41 smithi019 podman[112184]: 2024-03-26 02:10:41.625412479 +0000 UTC m=+5.588076584 container died 0401940d882b660b6928430c55ee7f32363ad9c83db4acf89a63bf56c9bf60b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:10:41.859 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:41 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:41.496+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:14.972652+0000 front 2024-03-26T02:10:14.972635+0000 (oldest deadline 2024-03-26T02:10:37.872709+0000) 2024-03-26T02:10:42.265 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:10:42 smithi019 podman[112184]: 2024-03-26 02:10:42.04053411 +0000 UTC m=+6.003198222 container cleanup 0401940d882b660b6928430c55ee7f32363ad9c83db4acf89a63bf56c9bf60b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-26T02:10:42.265 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:10:42 smithi019 bash[112184]: ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3 2024-03-26T02:10:42.359 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:42 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:42.094+0000 7fdda3bf5700 -1 osd.5 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.990521+0000 front 2024-03-26T02:10:15.990504+0000 (oldest deadline 2024-03-26T02:10:39.490509+0000) 2024-03-26T02:10:42.359 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:42 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:42.273+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:10:42.689 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:10:42 smithi019 podman[112212]: 2024-03-26 02:10:42.317105206 +0000 UTC m=+0.686222719 container remove 0401940d882b660b6928430c55ee7f32363ad9c83db4acf89a63bf56c9bf60b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124) 2024-03-26T02:10:42.859 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:10:42 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4[58030]: 2024-03-26T02:10:42.586+0000 7fbaac955700 -1 osd.4 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:16.801944+0000 front 2024-03-26T02:10:16.802185+0000 (oldest deadline 2024-03-26T02:10:42.102078+0000) 2024-03-26T02:10:42.860 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:42 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:42.509+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:14.972652+0000 front 2024-03-26T02:10:14.972635+0000 (oldest deadline 2024-03-26T02:10:37.872709+0000) 2024-03-26T02:10:42.954 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:10:42 smithi019 podman[112304]: 2024-03-26 02:10:42.592154245 +0000 UTC m=+0.018543653 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:10:42.955 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:10:42 smithi019 podman[112304]: 2024-03-26 02:10:42.709622114 +0000 UTC m=+0.136011524 container create 1d6bdf1fea707b81d0f64122b45f9fe02eb17f2043fafc77f5dada44f7a47dd0 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:10:42.955 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:10:42 smithi019 podman[112304]: 2024-03-26 02:10:42.953886681 +0000 UTC m=+0.380276088 container init 1d6bdf1fea707b81d0f64122b45f9fe02eb17f2043fafc77f5dada44f7a47dd0 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-26T02:10:43.255 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:10:42 smithi019 podman[112304]: 2024-03-26 02:10:42.95742093 +0000 UTC m=+0.383810341 container start 1d6bdf1fea707b81d0f64122b45f9fe02eb17f2043fafc77f5dada44f7a47dd0 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:10:43.256 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:10:43 smithi019 podman[112304]: 2024-03-26 02:10:43.070778301 +0000 UTC m=+0.497167716 container attach 1d6bdf1fea707b81d0f64122b45f9fe02eb17f2043fafc77f5dada44f7a47dd0 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS) 2024-03-26T02:10:43.359 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:43 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:43.074+0000 7fdda3bf5700 -1 osd.5 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.990521+0000 front 2024-03-26T02:10:15.990504+0000 (oldest deadline 2024-03-26T02:10:39.490509+0000) 2024-03-26T02:10:43.359 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:43 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:43.319+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:10:43.515 INFO:journalctl@ceph.osd.3.smithi019.stdout:Mar 26 02:10:43 smithi019 podman[112304]: 2024-03-26 02:10:43.255884626 +0000 UTC m=+0.682274047 container died 1d6bdf1fea707b81d0f64122b45f9fe02eb17f2043fafc77f5dada44f7a47dd0 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True) 2024-03-26T02:10:43.859 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:10:43 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4[58030]: 2024-03-26T02:10:43.595+0000 7fbaac955700 -1 osd.4 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:16.801944+0000 front 2024-03-26T02:10:16.802185+0000 (oldest deadline 2024-03-26T02:10:42.102078+0000) 2024-03-26T02:10:43.860 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:43 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:43.507+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:14.972652+0000 front 2024-03-26T02:10:14.972635+0000 (oldest deadline 2024-03-26T02:10:37.872709+0000) 2024-03-26T02:10:44.014 DEBUG:teuthology.orchestra.run.smithi019:> sudo pkill -f 'journalctl -f -n 0 -u ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.3.service' 2024-03-26T02:10:44.055 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:10:44.055 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-26T02:10:44.055 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-26T02:10:44.055 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.4 2024-03-26T02:10:44.359 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:10:44 smithi069 systemd[1]: Stopping Ceph osd.4 for 692607c6-eb12-11ee-95d0-87774f69a715... 2024-03-26T02:10:44.359 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:44 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:44.101+0000 7fdda3bf5700 -1 osd.5 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.990521+0000 front 2024-03-26T02:10:15.990504+0000 (oldest deadline 2024-03-26T02:10:39.490509+0000) 2024-03-26T02:10:44.360 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:44 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:44.301+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:10:44.859 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:44 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:44.474+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:14.972652+0000 front 2024-03-26T02:10:14.972635+0000 (oldest deadline 2024-03-26T02:10:37.872709+0000) 2024-03-26T02:10:44.859 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:10:44 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4[58030]: 2024-03-26T02:10:44.366+0000 7fbab0361700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:10:44.859 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:10:44 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4[58030]: 2024-03-26T02:10:44.366+0000 7fbab0361700 -1 osd.4 168 *** Got signal Terminated *** 2024-03-26T02:10:44.860 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:10:44 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4[58030]: 2024-03-26T02:10:44.366+0000 7fbab0361700 -1 osd.4 168 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:10:44.860 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:10:44 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4[58030]: 2024-03-26T02:10:44.615+0000 7fbaac955700 -1 osd.4 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:16.801944+0000 front 2024-03-26T02:10:16.802185+0000 (oldest deadline 2024-03-26T02:10:42.102078+0000) 2024-03-26T02:10:45.456 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:45 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:45.151+0000 7fdda3bf5700 -1 osd.5 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.990521+0000 front 2024-03-26T02:10:15.990504+0000 (oldest deadline 2024-03-26T02:10:39.490509+0000) 2024-03-26T02:10:45.456 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:45 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:45.295+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:10:45.859 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:10:45 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4[58030]: 2024-03-26T02:10:45.576+0000 7fbaac955700 -1 osd.4 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:16.801944+0000 front 2024-03-26T02:10:16.802185+0000 (oldest deadline 2024-03-26T02:10:42.102078+0000) 2024-03-26T02:10:45.859 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:45 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:45.455+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:14.972652+0000 front 2024-03-26T02:10:14.972635+0000 (oldest deadline 2024-03-26T02:10:37.872709+0000) 2024-03-26T02:10:46.436 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:46 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:46.265+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:10:46.436 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:46 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:46.133+0000 7fdda3bf5700 -1 osd.5 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.990521+0000 front 2024-03-26T02:10:15.990504+0000 (oldest deadline 2024-03-26T02:10:39.490509+0000) 2024-03-26T02:10:46.859 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:10:46 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4[58030]: 2024-03-26T02:10:46.545+0000 7fbaac955700 -1 osd.4 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:16.801944+0000 front 2024-03-26T02:10:16.802185+0000 (oldest deadline 2024-03-26T02:10:42.102078+0000) 2024-03-26T02:10:46.859 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:46 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:46.436+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:14.972652+0000 front 2024-03-26T02:10:14.972635+0000 (oldest deadline 2024-03-26T02:10:37.872709+0000) 2024-03-26T02:10:47.359 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:47 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:47.093+0000 7fdda3bf5700 -1 osd.5 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.990521+0000 front 2024-03-26T02:10:15.990504+0000 (oldest deadline 2024-03-26T02:10:39.490509+0000) 2024-03-26T02:10:47.360 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:47 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:47.228+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:10:47.859 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:10:47 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4[58030]: 2024-03-26T02:10:47.505+0000 7fbaac955700 -1 osd.4 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:16.801944+0000 front 2024-03-26T02:10:16.802185+0000 (oldest deadline 2024-03-26T02:10:42.102078+0000) 2024-03-26T02:10:47.859 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:47 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:47.398+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:14.972652+0000 front 2024-03-26T02:10:14.972635+0000 (oldest deadline 2024-03-26T02:10:37.872709+0000) 2024-03-26T02:10:47.859 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:47 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:47.398+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:21.373673+0000 front 2024-03-26T02:10:21.373552+0000 (oldest deadline 2024-03-26T02:10:47.273613+0000) 2024-03-26T02:10:48.359 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:48 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:48.080+0000 7fdda3bf5700 -1 osd.5 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.990521+0000 front 2024-03-26T02:10:15.990504+0000 (oldest deadline 2024-03-26T02:10:39.490509+0000) 2024-03-26T02:10:48.360 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:48 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:48.208+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:10:48.859 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:10:48 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4[58030]: 2024-03-26T02:10:48.512+0000 7fbaac955700 -1 osd.4 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:16.801944+0000 front 2024-03-26T02:10:16.802185+0000 (oldest deadline 2024-03-26T02:10:42.102078+0000) 2024-03-26T02:10:48.859 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:48 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:48.404+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:14.972652+0000 front 2024-03-26T02:10:14.972635+0000 (oldest deadline 2024-03-26T02:10:37.872709+0000) 2024-03-26T02:10:48.859 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:48 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:48.404+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:21.373673+0000 front 2024-03-26T02:10:21.373552+0000 (oldest deadline 2024-03-26T02:10:47.273613+0000) 2024-03-26T02:10:49.359 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:49 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:49.061+0000 7fdda3bf5700 -1 osd.5 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.990521+0000 front 2024-03-26T02:10:15.990504+0000 (oldest deadline 2024-03-26T02:10:39.490509+0000) 2024-03-26T02:10:49.359 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:49 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:49.061+0000 7fdda3bf5700 -1 osd.5 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.490974+0000 front 2024-03-26T02:10:19.491128+0000 (oldest deadline 2024-03-26T02:10:48.291576+0000) 2024-03-26T02:10:49.359 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:49 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:49.179+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:10:49.360 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:49 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:49.179+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.668533+0000 front 2024-03-26T02:10:19.668649+0000 (oldest deadline 2024-03-26T02:10:49.068832+0000) 2024-03-26T02:10:49.797 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:49 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:49.368+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:14.972652+0000 front 2024-03-26T02:10:14.972635+0000 (oldest deadline 2024-03-26T02:10:37.872709+0000) 2024-03-26T02:10:49.798 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:49 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:49.368+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:21.373673+0000 front 2024-03-26T02:10:21.373552+0000 (oldest deadline 2024-03-26T02:10:47.273613+0000) 2024-03-26T02:10:50.049 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:10:49 smithi069 podman[79520]: 2024-03-26 02:10:49.797623662 +0000 UTC m=+5.549549570 container died 6eb3b2ce99782f0593e80546272b83f504d3d49d8e09a8209a100a2545be3f11 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-26T02:10:50.359 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:10:50 smithi069 podman[79520]: 2024-03-26 02:10:50.217150653 +0000 UTC m=+5.969076565 container cleanup 6eb3b2ce99782f0593e80546272b83f504d3d49d8e09a8209a100a2545be3f11 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-26T02:10:50.359 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:10:50 smithi069 bash[79520]: ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4 2024-03-26T02:10:50.360 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:50 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:50.048+0000 7fdda3bf5700 -1 osd.5 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.990521+0000 front 2024-03-26T02:10:15.990504+0000 (oldest deadline 2024-03-26T02:10:39.490509+0000) 2024-03-26T02:10:50.360 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:50 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:50.048+0000 7fdda3bf5700 -1 osd.5 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.490974+0000 front 2024-03-26T02:10:19.491128+0000 (oldest deadline 2024-03-26T02:10:48.291576+0000) 2024-03-26T02:10:50.360 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:50 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:50.204+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:10:50.360 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:50 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:50.204+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.668533+0000 front 2024-03-26T02:10:19.668649+0000 (oldest deadline 2024-03-26T02:10:49.068832+0000) 2024-03-26T02:10:50.859 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:10:50 smithi069 podman[79569]: 2024-03-26 02:10:50.452185845 +0000 UTC m=+0.658140234 container remove 6eb3b2ce99782f0593e80546272b83f504d3d49d8e09a8209a100a2545be3f11 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-26T02:10:50.859 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:50 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:50.396+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:14.972652+0000 front 2024-03-26T02:10:14.972635+0000 (oldest deadline 2024-03-26T02:10:37.872709+0000) 2024-03-26T02:10:50.859 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:50 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:50.396+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:21.373673+0000 front 2024-03-26T02:10:21.373552+0000 (oldest deadline 2024-03-26T02:10:47.273613+0000) 2024-03-26T02:10:51.191 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:51 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:51.054+0000 7fdda3bf5700 -1 osd.5 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.990521+0000 front 2024-03-26T02:10:15.990504+0000 (oldest deadline 2024-03-26T02:10:39.490509+0000) 2024-03-26T02:10:51.191 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:51 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:51.054+0000 7fdda3bf5700 -1 osd.5 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.490974+0000 front 2024-03-26T02:10:19.491128+0000 (oldest deadline 2024-03-26T02:10:48.291576+0000) 2024-03-26T02:10:51.192 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:10:50 smithi069 podman[79659]: 2024-03-26 02:10:50.784302737 +0000 UTC m=+0.017461874 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:10:51.192 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:10:50 smithi069 podman[79659]: 2024-03-26 02:10:50.894846879 +0000 UTC m=+0.128006035 container create 464f89bd6753a9a193d2dec2d933f3138ab1f5f6c827bc7ae48430bf505e67aa (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4-deactivate, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:10:51.192 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:10:51 smithi069 podman[79659]: 2024-03-26 02:10:51.122941527 +0000 UTC m=+0.356100648 container init 464f89bd6753a9a193d2dec2d933f3138ab1f5f6c827bc7ae48430bf505e67aa (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:10:51.192 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:10:51 smithi069 podman[79659]: 2024-03-26 02:10:51.126452246 +0000 UTC m=+0.359611356 container start 464f89bd6753a9a193d2dec2d933f3138ab1f5f6c827bc7ae48430bf505e67aa (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-26T02:10:51.608 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:51 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:51.189+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:10:51.608 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:51 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:51.189+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.668533+0000 front 2024-03-26T02:10:19.668649+0000 (oldest deadline 2024-03-26T02:10:49.068832+0000) 2024-03-26T02:10:51.608 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:51 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:51.361+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:14.972652+0000 front 2024-03-26T02:10:14.972635+0000 (oldest deadline 2024-03-26T02:10:37.872709+0000) 2024-03-26T02:10:51.608 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:51 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:51.361+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:21.373673+0000 front 2024-03-26T02:10:21.373552+0000 (oldest deadline 2024-03-26T02:10:47.273613+0000) 2024-03-26T02:10:51.609 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:10:51 smithi069 podman[79659]: 2024-03-26 02:10:51.225059824 +0000 UTC m=+0.458218943 container attach 464f89bd6753a9a193d2dec2d933f3138ab1f5f6c827bc7ae48430bf505e67aa (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=squid-5e70124) 2024-03-26T02:10:51.609 INFO:journalctl@ceph.osd.4.smithi069.stdout:Mar 26 02:10:51 smithi069 podman[79659]: 2024-03-26 02:10:51.427665414 +0000 UTC m=+0.660824529 container died 464f89bd6753a9a193d2dec2d933f3138ab1f5f6c827bc7ae48430bf505e67aa (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-26T02:10:51.995 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.4.service' 2024-03-26T02:10:52.035 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:10:52.035 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-26T02:10:52.036 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-26T02:10:52.036 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.5 2024-03-26T02:10:52.359 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:52 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:52.178+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:10:52.359 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:52 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:52.178+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.668533+0000 front 2024-03-26T02:10:19.668649+0000 (oldest deadline 2024-03-26T02:10:49.068832+0000) 2024-03-26T02:10:52.360 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:52 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:52.102+0000 7fdda3bf5700 -1 osd.5 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.990521+0000 front 2024-03-26T02:10:15.990504+0000 (oldest deadline 2024-03-26T02:10:39.490509+0000) 2024-03-26T02:10:52.360 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:52 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:52.102+0000 7fdda3bf5700 -1 osd.5 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.490974+0000 front 2024-03-26T02:10:19.491128+0000 (oldest deadline 2024-03-26T02:10:48.291576+0000) 2024-03-26T02:10:52.360 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:52 smithi069 systemd[1]: Stopping Ceph osd.5 for 692607c6-eb12-11ee-95d0-87774f69a715... 2024-03-26T02:10:52.859 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:52 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:52.395+0000 7fddab609700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:10:52.859 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:52 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:52.395+0000 7fddab609700 -1 osd.5 168 *** Got signal Terminated *** 2024-03-26T02:10:52.860 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:52 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:52.395+0000 7fddab609700 -1 osd.5 168 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:10:52.860 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:52 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:52.411+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:14.972652+0000 front 2024-03-26T02:10:14.972635+0000 (oldest deadline 2024-03-26T02:10:37.872709+0000) 2024-03-26T02:10:52.860 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:52 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:52.411+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:21.373673+0000 front 2024-03-26T02:10:21.373552+0000 (oldest deadline 2024-03-26T02:10:47.273613+0000) 2024-03-26T02:10:53.459 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:53 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:53.151+0000 7fdda3bf5700 -1 osd.5 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.990521+0000 front 2024-03-26T02:10:15.990504+0000 (oldest deadline 2024-03-26T02:10:39.490509+0000) 2024-03-26T02:10:53.459 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:53 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:53.151+0000 7fdda3bf5700 -1 osd.5 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.490974+0000 front 2024-03-26T02:10:19.491128+0000 (oldest deadline 2024-03-26T02:10:48.291576+0000) 2024-03-26T02:10:53.459 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:53 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:53.189+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:10:53.460 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:53 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:53.189+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.668533+0000 front 2024-03-26T02:10:19.668649+0000 (oldest deadline 2024-03-26T02:10:49.068832+0000) 2024-03-26T02:10:53.859 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:53 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:53.458+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:14.972652+0000 front 2024-03-26T02:10:14.972635+0000 (oldest deadline 2024-03-26T02:10:37.872709+0000) 2024-03-26T02:10:53.859 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:53 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:53.458+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:21.373673+0000 front 2024-03-26T02:10:21.373552+0000 (oldest deadline 2024-03-26T02:10:47.273613+0000) 2024-03-26T02:10:54.483 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:54 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:54.155+0000 7fdda3bf5700 -1 osd.5 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.990521+0000 front 2024-03-26T02:10:15.990504+0000 (oldest deadline 2024-03-26T02:10:39.490509+0000) 2024-03-26T02:10:54.483 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:54 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:54.155+0000 7fdda3bf5700 -1 osd.5 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.490974+0000 front 2024-03-26T02:10:19.491128+0000 (oldest deadline 2024-03-26T02:10:48.291576+0000) 2024-03-26T02:10:54.483 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:54 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:54.168+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:10:54.483 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:54 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:54.168+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.668533+0000 front 2024-03-26T02:10:19.668649+0000 (oldest deadline 2024-03-26T02:10:49.068832+0000) 2024-03-26T02:10:54.859 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:54 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:54.482+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:14.972652+0000 front 2024-03-26T02:10:14.972635+0000 (oldest deadline 2024-03-26T02:10:37.872709+0000) 2024-03-26T02:10:54.859 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:54 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:54.482+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:21.373673+0000 front 2024-03-26T02:10:21.373552+0000 (oldest deadline 2024-03-26T02:10:47.273613+0000) 2024-03-26T02:10:54.859 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:54 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:54.482+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6822 osd.2 since back 2024-03-26T02:10:30.074978+0000 front 2024-03-26T02:10:30.074904+0000 (oldest deadline 2024-03-26T02:10:54.175023+0000) 2024-03-26T02:10:55.439 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:55 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:55.112+0000 7fdda3bf5700 -1 osd.5 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.990521+0000 front 2024-03-26T02:10:15.990504+0000 (oldest deadline 2024-03-26T02:10:39.490509+0000) 2024-03-26T02:10:55.440 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:55 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:55.112+0000 7fdda3bf5700 -1 osd.5 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.490974+0000 front 2024-03-26T02:10:19.491128+0000 (oldest deadline 2024-03-26T02:10:48.291576+0000) 2024-03-26T02:10:55.440 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:55 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:55.112+0000 7fdda3bf5700 -1 osd.5 168 heartbeat_check: no reply from 172.21.15.19:6822 osd.2 since back 2024-03-26T02:10:28.292012+0000 front 2024-03-26T02:10:28.292030+0000 (oldest deadline 2024-03-26T02:10:54.192192+0000) 2024-03-26T02:10:55.440 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:55 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:55.200+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:10:55.440 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:55 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:55.200+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.668533+0000 front 2024-03-26T02:10:19.668649+0000 (oldest deadline 2024-03-26T02:10:49.068832+0000) 2024-03-26T02:10:55.440 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:55 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:55.200+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6822 osd.2 since back 2024-03-26T02:10:29.069106+0000 front 2024-03-26T02:10:29.069163+0000 (oldest deadline 2024-03-26T02:10:54.969276+0000) 2024-03-26T02:10:55.859 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:55 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:55.438+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:14.972652+0000 front 2024-03-26T02:10:14.972635+0000 (oldest deadline 2024-03-26T02:10:37.872709+0000) 2024-03-26T02:10:55.859 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:55 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:55.438+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:21.373673+0000 front 2024-03-26T02:10:21.373552+0000 (oldest deadline 2024-03-26T02:10:47.273613+0000) 2024-03-26T02:10:55.859 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:55 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:55.438+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6822 osd.2 since back 2024-03-26T02:10:30.074978+0000 front 2024-03-26T02:10:30.074904+0000 (oldest deadline 2024-03-26T02:10:54.175023+0000) 2024-03-26T02:10:56.359 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:56 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:56.107+0000 7fdda3bf5700 -1 osd.5 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.990521+0000 front 2024-03-26T02:10:15.990504+0000 (oldest deadline 2024-03-26T02:10:39.490509+0000) 2024-03-26T02:10:56.359 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:56 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:56.107+0000 7fdda3bf5700 -1 osd.5 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.490974+0000 front 2024-03-26T02:10:19.491128+0000 (oldest deadline 2024-03-26T02:10:48.291576+0000) 2024-03-26T02:10:56.359 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:56 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:56.107+0000 7fdda3bf5700 -1 osd.5 168 heartbeat_check: no reply from 172.21.15.19:6822 osd.2 since back 2024-03-26T02:10:28.292012+0000 front 2024-03-26T02:10:28.292030+0000 (oldest deadline 2024-03-26T02:10:54.192192+0000) 2024-03-26T02:10:56.360 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:56 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:56.249+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:10:56.360 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:56 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:56.249+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.668533+0000 front 2024-03-26T02:10:19.668649+0000 (oldest deadline 2024-03-26T02:10:49.068832+0000) 2024-03-26T02:10:56.360 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:56 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:56.249+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6822 osd.2 since back 2024-03-26T02:10:29.069106+0000 front 2024-03-26T02:10:29.069163+0000 (oldest deadline 2024-03-26T02:10:54.969276+0000) 2024-03-26T02:10:56.859 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:56 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:56.470+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:14.972652+0000 front 2024-03-26T02:10:14.972635+0000 (oldest deadline 2024-03-26T02:10:37.872709+0000) 2024-03-26T02:10:56.859 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:56 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:56.470+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:21.373673+0000 front 2024-03-26T02:10:21.373552+0000 (oldest deadline 2024-03-26T02:10:47.273613+0000) 2024-03-26T02:10:56.859 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:56 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:56.470+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6822 osd.2 since back 2024-03-26T02:10:30.074978+0000 front 2024-03-26T02:10:30.074904+0000 (oldest deadline 2024-03-26T02:10:54.175023+0000) 2024-03-26T02:10:57.359 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:57 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:57.103+0000 7fdda3bf5700 -1 osd.5 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.990521+0000 front 2024-03-26T02:10:15.990504+0000 (oldest deadline 2024-03-26T02:10:39.490509+0000) 2024-03-26T02:10:57.359 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:57 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:57.103+0000 7fdda3bf5700 -1 osd.5 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.490974+0000 front 2024-03-26T02:10:19.491128+0000 (oldest deadline 2024-03-26T02:10:48.291576+0000) 2024-03-26T02:10:57.359 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:57 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5[60268]: 2024-03-26T02:10:57.103+0000 7fdda3bf5700 -1 osd.5 168 heartbeat_check: no reply from 172.21.15.19:6822 osd.2 since back 2024-03-26T02:10:28.292012+0000 front 2024-03-26T02:10:28.292030+0000 (oldest deadline 2024-03-26T02:10:54.192192+0000) 2024-03-26T02:10:57.360 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:57 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:57.233+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:10:57.360 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:57 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:57.233+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.668533+0000 front 2024-03-26T02:10:19.668649+0000 (oldest deadline 2024-03-26T02:10:49.068832+0000) 2024-03-26T02:10:57.360 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:57 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:57.233+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6822 osd.2 since back 2024-03-26T02:10:29.069106+0000 front 2024-03-26T02:10:29.069163+0000 (oldest deadline 2024-03-26T02:10:54.969276+0000) 2024-03-26T02:10:57.859 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:57 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:57.498+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:14.972652+0000 front 2024-03-26T02:10:14.972635+0000 (oldest deadline 2024-03-26T02:10:37.872709+0000) 2024-03-26T02:10:57.859 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:57 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:57.498+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:21.373673+0000 front 2024-03-26T02:10:21.373552+0000 (oldest deadline 2024-03-26T02:10:47.273613+0000) 2024-03-26T02:10:57.859 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:57 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:57.498+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6822 osd.2 since back 2024-03-26T02:10:30.074978+0000 front 2024-03-26T02:10:30.074904+0000 (oldest deadline 2024-03-26T02:10:54.175023+0000) 2024-03-26T02:10:58.188 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:57 smithi069 podman[79776]: 2024-03-26 02:10:57.919965757 +0000 UTC m=+5.644231516 container died fb31f0e68fb7a960d4760fc5788d1aa49546d767311574fddb8b7e9bb5e437f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-26T02:10:58.467 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:58 smithi069 podman[79776]: 2024-03-26 02:10:58.28724513 +0000 UTC m=+6.011510890 container cleanup fb31f0e68fb7a960d4760fc5788d1aa49546d767311574fddb8b7e9bb5e437f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-26T02:10:58.467 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:58 smithi069 bash[79776]: ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5 2024-03-26T02:10:58.467 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:58 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:58.261+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:10:58.467 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:58 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:58.261+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.668533+0000 front 2024-03-26T02:10:19.668649+0000 (oldest deadline 2024-03-26T02:10:49.068832+0000) 2024-03-26T02:10:58.467 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:58 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:58.261+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6822 osd.2 since back 2024-03-26T02:10:29.069106+0000 front 2024-03-26T02:10:29.069163+0000 (oldest deadline 2024-03-26T02:10:54.969276+0000) 2024-03-26T02:10:58.859 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:58 smithi069 podman[79794]: 2024-03-26 02:10:58.605200231 +0000 UTC m=+0.684718761 container remove fb31f0e68fb7a960d4760fc5788d1aa49546d767311574fddb8b7e9bb5e437f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-26T02:10:58.859 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:58 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:58.465+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:14.972652+0000 front 2024-03-26T02:10:14.972635+0000 (oldest deadline 2024-03-26T02:10:37.872709+0000) 2024-03-26T02:10:58.859 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:58 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:58.465+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:21.373673+0000 front 2024-03-26T02:10:21.373552+0000 (oldest deadline 2024-03-26T02:10:47.273613+0000) 2024-03-26T02:10:58.859 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:58 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:58.465+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6822 osd.2 since back 2024-03-26T02:10:30.074978+0000 front 2024-03-26T02:10:30.074904+0000 (oldest deadline 2024-03-26T02:10:54.175023+0000) 2024-03-26T02:10:59.234 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:58 smithi069 podman[79887]: 2024-03-26 02:10:58.851587591 +0000 UTC m=+0.018096250 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:10:59.234 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:58 smithi069 podman[79887]: 2024-03-26 02:10:58.956006404 +0000 UTC m=+0.122515087 container create 1271c710a70b78290289eea28ab45309be6b4f4b17a772b788d91ebd1d5f1bbc (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-26T02:10:59.234 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:59 smithi069 podman[79887]: 2024-03-26 02:10:59.158388957 +0000 UTC m=+0.324897598 container init 1271c710a70b78290289eea28ab45309be6b4f4b17a772b788d91ebd1d5f1bbc (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:10:59.234 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:59 smithi069 podman[79887]: 2024-03-26 02:10:59.161606077 +0000 UTC m=+0.328114722 container start 1271c710a70b78290289eea28ab45309be6b4f4b17a772b788d91ebd1d5f1bbc (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-26T02:10:59.234 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:59 smithi069 podman[79887]: 2024-03-26 02:10:59.233130833 +0000 UTC m=+0.399639501 container attach 1271c710a70b78290289eea28ab45309be6b4f4b17a772b788d91ebd1d5f1bbc (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-26T02:10:59.609 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:59 smithi069 podman[79887]: 2024-03-26 02:10:59.451004445 +0000 UTC m=+0.617513098 container died 1271c710a70b78290289eea28ab45309be6b4f4b17a772b788d91ebd1d5f1bbc (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-26T02:10:59.610 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:59 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:59.279+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:10:59.610 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:59 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:59.279+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.668533+0000 front 2024-03-26T02:10:19.668649+0000 (oldest deadline 2024-03-26T02:10:49.068832+0000) 2024-03-26T02:10:59.610 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:10:59 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:10:59.279+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6822 osd.2 since back 2024-03-26T02:10:29.069106+0000 front 2024-03-26T02:10:29.069163+0000 (oldest deadline 2024-03-26T02:10:54.969276+0000) 2024-03-26T02:10:59.610 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:59 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:59.480+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:14.972652+0000 front 2024-03-26T02:10:14.972635+0000 (oldest deadline 2024-03-26T02:10:37.872709+0000) 2024-03-26T02:10:59.610 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:59 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:59.480+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:21.373673+0000 front 2024-03-26T02:10:21.373552+0000 (oldest deadline 2024-03-26T02:10:47.273613+0000) 2024-03-26T02:10:59.610 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:10:59 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:10:59.480+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6822 osd.2 since back 2024-03-26T02:10:30.074978+0000 front 2024-03-26T02:10:30.074904+0000 (oldest deadline 2024-03-26T02:10:54.175023+0000) 2024-03-26T02:10:59.988 INFO:journalctl@ceph.osd.5.smithi069.stdout:Mar 26 02:10:59 smithi069 podman[79914]: 2024-03-26 02:10:59.969942641 +0000 UTC m=+0.513359146 container remove 1271c710a70b78290289eea28ab45309be6b4f4b17a772b788d91ebd1d5f1bbc (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-26T02:10:59.992 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.5.service' 2024-03-26T02:11:00.037 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:11:00.038 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-26T02:11:00.038 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-26T02:11:00.038 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.6 2024-03-26T02:11:00.275 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:11:00 smithi069 systemd[1]: Stopping Ceph osd.6 for 692607c6-eb12-11ee-95d0-87774f69a715... 2024-03-26T02:11:00.609 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:11:00 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:11:00.364+0000 7f9370d21700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:11:00.609 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:11:00 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:11:00.364+0000 7f9370d21700 -1 osd.6 168 *** Got signal Terminated *** 2024-03-26T02:11:00.609 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:11:00 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:11:00.364+0000 7f9370d21700 -1 osd.6 168 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:11:00.610 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:11:00 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:11:00.494+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:14.972652+0000 front 2024-03-26T02:10:14.972635+0000 (oldest deadline 2024-03-26T02:10:37.872709+0000) 2024-03-26T02:11:00.610 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:11:00 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:11:00.494+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:21.373673+0000 front 2024-03-26T02:10:21.373552+0000 (oldest deadline 2024-03-26T02:10:47.273613+0000) 2024-03-26T02:11:00.610 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:11:00 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:11:00.494+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6822 osd.2 since back 2024-03-26T02:10:30.074978+0000 front 2024-03-26T02:10:30.074904+0000 (oldest deadline 2024-03-26T02:10:54.175023+0000) 2024-03-26T02:11:00.610 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:00 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:00.273+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:11:00.610 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:00 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:00.273+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.668533+0000 front 2024-03-26T02:10:19.668649+0000 (oldest deadline 2024-03-26T02:10:49.068832+0000) 2024-03-26T02:11:00.610 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:00 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:00.273+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6822 osd.2 since back 2024-03-26T02:10:29.069106+0000 front 2024-03-26T02:10:29.069163+0000 (oldest deadline 2024-03-26T02:10:54.969276+0000) 2024-03-26T02:11:01.609 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:11:01 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:11:01.446+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:14.972652+0000 front 2024-03-26T02:10:14.972635+0000 (oldest deadline 2024-03-26T02:10:37.872709+0000) 2024-03-26T02:11:01.609 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:11:01 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:11:01.446+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:21.373673+0000 front 2024-03-26T02:10:21.373552+0000 (oldest deadline 2024-03-26T02:10:47.273613+0000) 2024-03-26T02:11:01.609 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:11:01 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:11:01.446+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6822 osd.2 since back 2024-03-26T02:10:30.074978+0000 front 2024-03-26T02:10:30.074904+0000 (oldest deadline 2024-03-26T02:10:54.175023+0000) 2024-03-26T02:11:01.610 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:01 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:01.310+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:11:01.610 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:01 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:01.310+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.668533+0000 front 2024-03-26T02:10:19.668649+0000 (oldest deadline 2024-03-26T02:10:49.068832+0000) 2024-03-26T02:11:01.610 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:01 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:01.310+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6822 osd.2 since back 2024-03-26T02:10:29.069106+0000 front 2024-03-26T02:10:29.069163+0000 (oldest deadline 2024-03-26T02:10:54.969276+0000) 2024-03-26T02:11:02.609 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:02 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:02.280+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:11:02.609 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:02 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:02.280+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.668533+0000 front 2024-03-26T02:10:19.668649+0000 (oldest deadline 2024-03-26T02:10:49.068832+0000) 2024-03-26T02:11:02.610 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:02 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:02.280+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6822 osd.2 since back 2024-03-26T02:10:29.069106+0000 front 2024-03-26T02:10:29.069163+0000 (oldest deadline 2024-03-26T02:10:54.969276+0000) 2024-03-26T02:11:02.610 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:02 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:02.280+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6830 osd.3 since back 2024-03-26T02:10:37.870336+0000 front 2024-03-26T02:10:37.870266+0000 (oldest deadline 2024-03-26T02:11:01.970297+0000) 2024-03-26T02:11:02.610 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:11:02 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:11:02.471+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:14.972652+0000 front 2024-03-26T02:10:14.972635+0000 (oldest deadline 2024-03-26T02:10:37.872709+0000) 2024-03-26T02:11:02.610 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:11:02 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:11:02.471+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:21.373673+0000 front 2024-03-26T02:10:21.373552+0000 (oldest deadline 2024-03-26T02:10:47.273613+0000) 2024-03-26T02:11:02.610 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:11:02 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:11:02.471+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6822 osd.2 since back 2024-03-26T02:10:30.074978+0000 front 2024-03-26T02:10:30.074904+0000 (oldest deadline 2024-03-26T02:10:54.175023+0000) 2024-03-26T02:11:02.610 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:11:02 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:11:02.471+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6830 osd.3 since back 2024-03-26T02:10:35.275859+0000 front 2024-03-26T02:10:35.275843+0000 (oldest deadline 2024-03-26T02:11:02.276158+0000) 2024-03-26T02:11:03.609 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:03 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:03.279+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:11:03.610 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:03 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:03.279+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.668533+0000 front 2024-03-26T02:10:19.668649+0000 (oldest deadline 2024-03-26T02:10:49.068832+0000) 2024-03-26T02:11:03.610 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:03 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:03.279+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6822 osd.2 since back 2024-03-26T02:10:29.069106+0000 front 2024-03-26T02:10:29.069163+0000 (oldest deadline 2024-03-26T02:10:54.969276+0000) 2024-03-26T02:11:03.610 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:03 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:03.279+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6830 osd.3 since back 2024-03-26T02:10:37.870336+0000 front 2024-03-26T02:10:37.870266+0000 (oldest deadline 2024-03-26T02:11:01.970297+0000) 2024-03-26T02:11:03.610 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:11:03 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:11:03.512+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:14.972652+0000 front 2024-03-26T02:10:14.972635+0000 (oldest deadline 2024-03-26T02:10:37.872709+0000) 2024-03-26T02:11:03.610 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:11:03 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:11:03.512+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:21.373673+0000 front 2024-03-26T02:10:21.373552+0000 (oldest deadline 2024-03-26T02:10:47.273613+0000) 2024-03-26T02:11:03.610 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:11:03 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:11:03.512+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6822 osd.2 since back 2024-03-26T02:10:30.074978+0000 front 2024-03-26T02:10:30.074904+0000 (oldest deadline 2024-03-26T02:10:54.175023+0000) 2024-03-26T02:11:03.610 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:11:03 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:11:03.512+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6830 osd.3 since back 2024-03-26T02:10:35.275859+0000 front 2024-03-26T02:10:35.275843+0000 (oldest deadline 2024-03-26T02:11:02.276158+0000) 2024-03-26T02:11:04.609 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:11:04 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:11:04.509+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:14.972652+0000 front 2024-03-26T02:10:14.972635+0000 (oldest deadline 2024-03-26T02:10:37.872709+0000) 2024-03-26T02:11:04.609 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:11:04 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:11:04.509+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:21.373673+0000 front 2024-03-26T02:10:21.373552+0000 (oldest deadline 2024-03-26T02:10:47.273613+0000) 2024-03-26T02:11:04.609 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:11:04 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:11:04.509+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6822 osd.2 since back 2024-03-26T02:10:30.074978+0000 front 2024-03-26T02:10:30.074904+0000 (oldest deadline 2024-03-26T02:10:54.175023+0000) 2024-03-26T02:11:04.610 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:11:04 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6[62516]: 2024-03-26T02:11:04.509+0000 7f936d315700 -1 osd.6 168 heartbeat_check: no reply from 172.21.15.19:6830 osd.3 since back 2024-03-26T02:10:35.275859+0000 front 2024-03-26T02:10:35.275843+0000 (oldest deadline 2024-03-26T02:11:02.276158+0000) 2024-03-26T02:11:04.610 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:04 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:04.299+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:11:04.610 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:04 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:04.299+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.668533+0000 front 2024-03-26T02:10:19.668649+0000 (oldest deadline 2024-03-26T02:10:49.068832+0000) 2024-03-26T02:11:04.610 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:04 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:04.299+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6822 osd.2 since back 2024-03-26T02:10:29.069106+0000 front 2024-03-26T02:10:29.069163+0000 (oldest deadline 2024-03-26T02:10:54.969276+0000) 2024-03-26T02:11:04.610 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:04 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:04.299+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6830 osd.3 since back 2024-03-26T02:10:37.870336+0000 front 2024-03-26T02:10:37.870266+0000 (oldest deadline 2024-03-26T02:11:01.970297+0000) 2024-03-26T02:11:05.609 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:05 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:05.312+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:11:05.609 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:05 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:05.312+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.668533+0000 front 2024-03-26T02:10:19.668649+0000 (oldest deadline 2024-03-26T02:10:49.068832+0000) 2024-03-26T02:11:05.609 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:05 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:05.312+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6822 osd.2 since back 2024-03-26T02:10:29.069106+0000 front 2024-03-26T02:10:29.069163+0000 (oldest deadline 2024-03-26T02:10:54.969276+0000) 2024-03-26T02:11:05.609 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:05 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:05.312+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6830 osd.3 since back 2024-03-26T02:10:37.870336+0000 front 2024-03-26T02:10:37.870266+0000 (oldest deadline 2024-03-26T02:11:01.970297+0000) 2024-03-26T02:11:06.057 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:11:05 smithi069 podman[79994]: 2024-03-26 02:11:05.74443472 +0000 UTC m=+5.479054367 container died 924e2a35053c1da0422a9d75ce49528904f8e881b1577fa463a94ea57172ebdb (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2) 2024-03-26T02:11:06.331 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:11:06 smithi069 podman[79994]: 2024-03-26 02:11:06.206200513 +0000 UTC m=+5.940820161 container cleanup 924e2a35053c1da0422a9d75ce49528904f8e881b1577fa463a94ea57172ebdb (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True) 2024-03-26T02:11:06.331 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:11:06 smithi069 bash[79994]: ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6 2024-03-26T02:11:06.609 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:11:06 smithi069 podman[80011]: 2024-03-26 02:11:06.499343503 +0000 UTC m=+0.747873761 container remove 924e2a35053c1da0422a9d75ce49528904f8e881b1577fa463a94ea57172ebdb (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-26T02:11:06.609 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:06 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:06.329+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:11:06.610 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:06 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:06.329+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.668533+0000 front 2024-03-26T02:10:19.668649+0000 (oldest deadline 2024-03-26T02:10:49.068832+0000) 2024-03-26T02:11:06.610 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:06 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:06.329+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6822 osd.2 since back 2024-03-26T02:10:29.069106+0000 front 2024-03-26T02:10:29.069163+0000 (oldest deadline 2024-03-26T02:10:54.969276+0000) 2024-03-26T02:11:06.610 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:06 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:06.329+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6830 osd.3 since back 2024-03-26T02:10:37.870336+0000 front 2024-03-26T02:10:37.870266+0000 (oldest deadline 2024-03-26T02:11:01.970297+0000) 2024-03-26T02:11:07.109 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:11:06 smithi069 podman[80106]: 2024-03-26 02:11:06.718905864 +0000 UTC m=+0.017986346 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:11:07.109 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:11:06 smithi069 podman[80106]: 2024-03-26 02:11:06.816669231 +0000 UTC m=+0.115749697 container create 1e7266cf99941deed8a632a12ddc8ba17f70197c056de9a7146830ea95e27002 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:11:07.110 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:11:07 smithi069 podman[80106]: 2024-03-26 02:11:07.03619024 +0000 UTC m=+0.335270714 container init 1e7266cf99941deed8a632a12ddc8ba17f70197c056de9a7146830ea95e27002 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-5e70124, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:11:07.110 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:11:07 smithi069 podman[80106]: 2024-03-26 02:11:07.039498222 +0000 UTC m=+0.338578691 container start 1e7266cf99941deed8a632a12ddc8ba17f70197c056de9a7146830ea95e27002 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-26T02:11:07.609 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:11:07 smithi069 podman[80106]: 2024-03-26 02:11:07.110820683 +0000 UTC m=+0.409901168 container attach 1e7266cf99941deed8a632a12ddc8ba17f70197c056de9a7146830ea95e27002 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319) 2024-03-26T02:11:07.609 INFO:journalctl@ceph.osd.6.smithi069.stdout:Mar 26 02:11:07 smithi069 podman[80106]: 2024-03-26 02:11:07.332052877 +0000 UTC m=+0.631133345 container died 1e7266cf99941deed8a632a12ddc8ba17f70197c056de9a7146830ea95e27002 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-5e70124, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:11:07.609 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:07 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:07.310+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:11:07.609 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:07 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:07.310+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.668533+0000 front 2024-03-26T02:10:19.668649+0000 (oldest deadline 2024-03-26T02:10:49.068832+0000) 2024-03-26T02:11:07.609 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:07 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:07.310+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6822 osd.2 since back 2024-03-26T02:10:29.069106+0000 front 2024-03-26T02:10:29.069163+0000 (oldest deadline 2024-03-26T02:10:54.969276+0000) 2024-03-26T02:11:07.610 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:07 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:07.310+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6830 osd.3 since back 2024-03-26T02:10:37.870336+0000 front 2024-03-26T02:10:37.870266+0000 (oldest deadline 2024-03-26T02:11:01.970297+0000) 2024-03-26T02:11:07.919 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.6.service' 2024-03-26T02:11:07.959 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:11:07.959 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-26T02:11:07.959 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-26T02:11:07.959 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.7 2024-03-26T02:11:08.284 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:08 smithi069 systemd[1]: Stopping Ceph osd.7 for 692607c6-eb12-11ee-95d0-87774f69a715... 2024-03-26T02:11:08.609 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:08 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:08.283+0000 7f0244525700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:11:08.609 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:08 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:08.283+0000 7f0244525700 -1 osd.7 168 *** Got signal Terminated *** 2024-03-26T02:11:08.609 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:08 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:08.283+0000 7f0244525700 -1 osd.7 168 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:11:08.610 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:08 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:08.354+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:11:08.610 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:08 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:08.354+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.668533+0000 front 2024-03-26T02:10:19.668649+0000 (oldest deadline 2024-03-26T02:10:49.068832+0000) 2024-03-26T02:11:08.610 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:08 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:08.354+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6822 osd.2 since back 2024-03-26T02:10:29.069106+0000 front 2024-03-26T02:10:29.069163+0000 (oldest deadline 2024-03-26T02:10:54.969276+0000) 2024-03-26T02:11:08.610 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:08 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:08.354+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6830 osd.3 since back 2024-03-26T02:10:37.870336+0000 front 2024-03-26T02:10:37.870266+0000 (oldest deadline 2024-03-26T02:11:01.970297+0000) 2024-03-26T02:11:09.609 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:09 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:09.346+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:11:09.609 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:09 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:09.346+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.668533+0000 front 2024-03-26T02:10:19.668649+0000 (oldest deadline 2024-03-26T02:10:49.068832+0000) 2024-03-26T02:11:09.609 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:09 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:09.346+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6822 osd.2 since back 2024-03-26T02:10:29.069106+0000 front 2024-03-26T02:10:29.069163+0000 (oldest deadline 2024-03-26T02:10:54.969276+0000) 2024-03-26T02:11:09.609 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:09 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:09.346+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6830 osd.3 since back 2024-03-26T02:10:37.870336+0000 front 2024-03-26T02:10:37.870266+0000 (oldest deadline 2024-03-26T02:11:01.970297+0000) 2024-03-26T02:11:10.859 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:10 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:10.390+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:11:10.859 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:10 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:10.390+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.668533+0000 front 2024-03-26T02:10:19.668649+0000 (oldest deadline 2024-03-26T02:10:49.068832+0000) 2024-03-26T02:11:10.859 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:10 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:10.390+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6822 osd.2 since back 2024-03-26T02:10:29.069106+0000 front 2024-03-26T02:10:29.069163+0000 (oldest deadline 2024-03-26T02:10:54.969276+0000) 2024-03-26T02:11:10.859 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:10 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:10.390+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6830 osd.3 since back 2024-03-26T02:10:37.870336+0000 front 2024-03-26T02:10:37.870266+0000 (oldest deadline 2024-03-26T02:11:01.970297+0000) 2024-03-26T02:11:10.860 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:10 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:10.390+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.69:6804 osd.4 since back 2024-03-26T02:10:45.971472+0000 front 2024-03-26T02:10:45.971582+0000 (oldest deadline 2024-03-26T02:11:10.071624+0000) 2024-03-26T02:11:11.859 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:11 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:11.371+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:11:11.859 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:11 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:11.371+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.668533+0000 front 2024-03-26T02:10:19.668649+0000 (oldest deadline 2024-03-26T02:10:49.068832+0000) 2024-03-26T02:11:11.859 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:11 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:11.371+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6822 osd.2 since back 2024-03-26T02:10:29.069106+0000 front 2024-03-26T02:10:29.069163+0000 (oldest deadline 2024-03-26T02:10:54.969276+0000) 2024-03-26T02:11:11.859 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:11 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:11.371+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6830 osd.3 since back 2024-03-26T02:10:37.870336+0000 front 2024-03-26T02:10:37.870266+0000 (oldest deadline 2024-03-26T02:11:01.970297+0000) 2024-03-26T02:11:11.859 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:11 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:11.371+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.69:6804 osd.4 since back 2024-03-26T02:10:45.971472+0000 front 2024-03-26T02:10:45.971582+0000 (oldest deadline 2024-03-26T02:11:10.071624+0000) 2024-03-26T02:11:12.863 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:12 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:12.419+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-03-26T02:10:15.667231+0000 front 2024-03-26T02:10:15.667530+0000 (oldest deadline 2024-03-26T02:10:38.567380+0000) 2024-03-26T02:11:12.863 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:12 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:12.419+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6814 osd.1 since back 2024-03-26T02:10:19.668533+0000 front 2024-03-26T02:10:19.668649+0000 (oldest deadline 2024-03-26T02:10:49.068832+0000) 2024-03-26T02:11:12.863 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:12 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:12.419+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6822 osd.2 since back 2024-03-26T02:10:29.069106+0000 front 2024-03-26T02:10:29.069163+0000 (oldest deadline 2024-03-26T02:10:54.969276+0000) 2024-03-26T02:11:12.863 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:12 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:12.419+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.19:6830 osd.3 since back 2024-03-26T02:10:37.870336+0000 front 2024-03-26T02:10:37.870266+0000 (oldest deadline 2024-03-26T02:11:01.970297+0000) 2024-03-26T02:11:12.863 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:12 smithi069 ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7[64757]: 2024-03-26T02:11:12.419+0000 7f0240b19700 -1 osd.7 168 heartbeat_check: no reply from 172.21.15.69:6804 osd.4 since back 2024-03-26T02:10:45.971472+0000 front 2024-03-26T02:10:45.971582+0000 (oldest deadline 2024-03-26T02:11:10.071624+0000) 2024-03-26T02:11:14.176 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:13 smithi069 podman[80243]: 2024-03-26 02:11:13.914042131 +0000 UTC m=+5.736808073 container died d8052b0a18d6edf22af2ab6e4a52aa9f5d56b69ad64c8a2ccbffbacd804f473b (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-26T02:11:14.609 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:14 smithi069 podman[80243]: 2024-03-26 02:11:14.366818675 +0000 UTC m=+6.189584626 container cleanup d8052b0a18d6edf22af2ab6e4a52aa9f5d56b69ad64c8a2ccbffbacd804f473b (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-26T02:11:14.609 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:14 smithi069 bash[80243]: ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7 2024-03-26T02:11:14.935 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:14 smithi069 podman[80275]: 2024-03-26 02:11:14.651452648 +0000 UTC m=+0.734709300 container remove d8052b0a18d6edf22af2ab6e4a52aa9f5d56b69ad64c8a2ccbffbacd804f473b (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:11:15.229 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:14 smithi069 podman[80366]: 2024-03-26 02:11:14.935408521 +0000 UTC m=+0.097191248 container create 7f7ac36565fba2a7bd2f8eaafa24c83e2779ac5a457f0377fce2c241d000fa14 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-26T02:11:15.229 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:14 smithi069 podman[80366]: 2024-03-26 02:11:14.856301716 +0000 UTC m=+0.018084443 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:11:15.230 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:15 smithi069 podman[80366]: 2024-03-26 02:11:15.229499011 +0000 UTC m=+0.391281738 container init 7f7ac36565fba2a7bd2f8eaafa24c83e2779ac5a457f0377fce2c241d000fa14 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-5e70124) 2024-03-26T02:11:15.540 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:15 smithi069 podman[80366]: 2024-03-26 02:11:15.232861092 +0000 UTC m=+0.394643820 container start 7f7ac36565fba2a7bd2f8eaafa24c83e2779ac5a457f0377fce2c241d000fa14 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-26T02:11:15.540 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:15 smithi069 podman[80366]: 2024-03-26 02:11:15.312758454 +0000 UTC m=+0.474541200 container attach 7f7ac36565fba2a7bd2f8eaafa24c83e2779ac5a457f0377fce2c241d000fa14 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-26T02:11:15.816 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:15 smithi069 podman[80366]: 2024-03-26 02:11:15.540350927 +0000 UTC m=+0.702133647 container died 7f7ac36565fba2a7bd2f8eaafa24c83e2779ac5a457f0377fce2c241d000fa14 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:11:16.055 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.7.service' 2024-03-26T02:11:16.081 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:16 smithi069 podman[80393]: 2024-03-26 02:11:16.032585882 +0000 UTC m=+0.486697088 container remove 7f7ac36565fba2a7bd2f8eaafa24c83e2779ac5a457f0377fce2c241d000fa14 (image=quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-692607c6-eb12-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True) 2024-03-26T02:11:16.081 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:16 smithi069 systemd[1]: ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.7.service: Deactivated successfully. 2024-03-26T02:11:16.082 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:16 smithi069 systemd[1]: Stopped Ceph osd.7 for 692607c6-eb12-11ee-95d0-87774f69a715. 2024-03-26T02:11:16.082 INFO:journalctl@ceph.osd.7.smithi069.stdout:Mar 26 02:11:16 smithi069 systemd[1]: ceph-692607c6-eb12-11ee-95d0-87774f69a715@osd.7.service: Consumed 11.002s CPU time. 2024-03-26T02:11:16.097 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:11:16.097 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-26T02:11:16.098 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 692607c6-eb12-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T02:14:16.421 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 692607c6-eb12-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T02:14:35.465 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T02:14:35.505 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T02:14:35.535 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-26T02:14:35.537 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621920/remote/smithi019/crash 2024-03-26T02:14:35.538 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/crash -- . 2024-03-26T02:14:35.595 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T02:14:35.595 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now 2024-03-26T02:14:35.597 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621920/remote/smithi069/crash 2024-03-26T02:14:35.598 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/crash -- . 2024-03-26T02:14:35.632 INFO:teuthology.orchestra.run.smithi069.stderr:tar: /var/lib/ceph/692607c6-eb12-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T02:14:35.633 INFO:teuthology.orchestra.run.smithi069.stderr:tar: Error is not recoverable: exiting now 2024-03-26T02:14:35.634 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-26T02:14:35.634 DEBUG:teuthology.orchestra.run.smithi019:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-26T02:14:35.665 INFO:teuthology.orchestra.run.smithi019.stdout:2024-03-26T01:50:42.933185+0000 mon.smithi019 (mon.0) 504 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-26T02:14:35.666 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-26T02:14:35.666 DEBUG:teuthology.orchestra.run.smithi019:> sudo egrep '\[SEC\]' /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-26T02:14:35.736 DEBUG:teuthology.orchestra.run.smithi019:> sudo egrep '\[ERR\]' /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-26T02:14:35.807 DEBUG:teuthology.orchestra.run.smithi019:> sudo egrep '\[WRN\]' /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-26T02:14:35.879 INFO:teuthology.orchestra.run.smithi019.stdout:2024-03-26T01:50:42.933185+0000 mon.smithi019 (mon.0) 504 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-26T02:14:35.879 INFO:tasks.cephadm:Compressing logs... 2024-03-26T02:14:35.880 DEBUG:teuthology.orchestra.run.smithi019:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T02:14:35.922 DEBUG:teuthology.orchestra.run.smithi069:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T02:14:35.947 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-26T02:14:35.947 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T02:14:35.947 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-mds.foofs.smithi019.lmryom.log 2024-03-26T02:14:35.947 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T02:14:35.948 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-mds.foofs.smithi019.lmryom.log: gzip -5 --verbose -- /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-osd.1.log 2024-03-26T02:14:35.953 INFO:teuthology.orchestra.run.smithi069.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T02:14:35.953 INFO:teuthology.orchestra.run.smithi069.stderr:: No such file or directory 2024-03-26T02:14:35.953 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-osd.6.log 2024-03-26T02:14:35.953 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-mon.smithi069.log 2024-03-26T02:14:35.953 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-mgr.smithi069.wpbewu.log 2024-03-26T02:14:35.953 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-mon.smithi069.log: gzip -5 --verbose -- /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T02:14:35.953 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-osd.4.log 2024-03-26T02:14:35.953 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-mgr.smithi069.wpbewu.log: /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-osd.7.log 2024-03-26T02:14:35.953 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-osd.5.log 2024-03-26T02:14:35.954 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-mds.foofs.smithi069.spgwgo.log 2024-03-26T02:14:35.955 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-osd.3.log 2024-03-26T02:14:35.955 INFO:teuthology.orchestra.run.smithi019.stderr: 89.3% -- replaced with /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-mds.foofs.smithi019.lmryom.log.gz 2024-03-26T02:14:35.955 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-mgr.smithi019.ixuvdm.log 2024-03-26T02:14:35.955 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-mon.smithi019.log 2024-03-26T02:14:35.955 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-osd.3.log: /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-mgr.smithi019.ixuvdm.log: gzip -5 --verbose -- /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-osd.0.log 2024-03-26T02:14:35.955 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-mon.smithi019.log: gzip -5 --verbose -- /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T02:14:35.956 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph.log 2024-03-26T02:14:35.956 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph.cephadm.log: 85.7% -- replaced with /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T02:14:35.956 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-osd.2.log 2024-03-26T02:14:35.956 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph.log: 91.5% -- replaced with /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T02:14:35.956 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T02:14:35.958 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-osd.5.log: 91.6% -- replaced with /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T02:14:35.958 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T02:14:35.958 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-mds.foofs.smithi069.spgwgo.log: 90.2% 87.7% -- replaced with /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-mds.foofs.smithi069.spgwgo.log.gz 2024-03-26T02:14:35.958 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T02:14:35.959 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph.log 2024-03-26T02:14:35.959 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T02:14:35.959 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph.log: 85.7% -- replaced with /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T02:14:35.959 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-osd.2.log: 87.5% -- replaced with /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T02:14:35.961 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-volume.log: 87.5% -- replaced with /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T02:14:35.967 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-volume.log: 90.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T02:14:35.973 INFO:teuthology.orchestra.run.smithi069.stderr: 91.5% -- replaced with /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T02:14:35.974 INFO:teuthology.orchestra.run.smithi019.stderr: 91.3% -- replaced with /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T02:14:36.009 INFO:teuthology.orchestra.run.smithi069.stderr: 89.2% -- replaced with /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-mgr.smithi069.wpbewu.log.gz 2024-03-26T02:14:36.236 INFO:teuthology.orchestra.run.smithi019.stderr: 88.3% -- replaced with /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-mgr.smithi019.ixuvdm.log.gz 2024-03-26T02:14:36.450 INFO:teuthology.orchestra.run.smithi069.stderr: 92.6% -- replaced with /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-mon.smithi069.log.gz 2024-03-26T02:14:37.444 INFO:teuthology.orchestra.run.smithi019.stderr: 92.0% -- replaced with /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-mon.smithi019.log.gz 2024-03-26T02:14:48.841 INFO:teuthology.orchestra.run.smithi019.stderr: 94.5% -- replaced with /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-osd.2.log.gz 2024-03-26T02:14:51.235 INFO:teuthology.orchestra.run.smithi069.stderr: 93.9% -- replaced with /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-osd.4.log.gz 2024-03-26T02:14:53.228 INFO:teuthology.orchestra.run.smithi069.stderr: 94.2% -- replaced with /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-osd.6.log.gz 2024-03-26T02:14:54.481 INFO:teuthology.orchestra.run.smithi019.stderr: 94.1% -- replaced with /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-osd.0.log.gz 2024-03-26T02:14:54.920 INFO:teuthology.orchestra.run.smithi019.stderr: 94.2% -- replaced with /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-osd.1.log.gz 2024-03-26T02:14:57.156 INFO:teuthology.orchestra.run.smithi069.stderr: 94.1% -- replaced with /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-osd.5.log.gz 2024-03-26T02:14:57.750 INFO:teuthology.orchestra.run.smithi019.stderr: 94.1% -- replaced with /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-osd.3.log.gz 2024-03-26T02:14:57.752 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-26T02:14:57.753 INFO:teuthology.orchestra.run.smithi019.stderr:real 0m21.818s 2024-03-26T02:14:57.753 INFO:teuthology.orchestra.run.smithi019.stderr:user 1m11.367s 2024-03-26T02:14:57.753 INFO:teuthology.orchestra.run.smithi019.stderr:sys 0m2.548s 2024-03-26T02:15:02.459 INFO:teuthology.orchestra.run.smithi069.stderr: 94.0% -- replaced with /var/log/ceph/692607c6-eb12-11ee-95d0-87774f69a715/ceph-osd.7.log.gz 2024-03-26T02:15:02.460 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-03-26T02:15:02.460 INFO:teuthology.orchestra.run.smithi069.stderr:real 0m26.524s 2024-03-26T02:15:02.460 INFO:teuthology.orchestra.run.smithi069.stderr:user 1m17.692s 2024-03-26T02:15:02.460 INFO:teuthology.orchestra.run.smithi069.stderr:sys 0m3.050s 2024-03-26T02:15:02.461 INFO:tasks.cephadm:Archiving logs... 2024-03-26T02:15:02.461 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621920/remote/smithi019/log 2024-03-26T02:15:02.462 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T02:15:12.300 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621920/remote/smithi069/log 2024-03-26T02:15:12.301 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T02:15:23.389 INFO:tasks.cephadm:Removing cluster... 2024-03-26T02:15:23.389 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 692607c6-eb12-11ee-95d0-87774f69a715 --force 2024-03-26T02:15:23.769 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 692607c6-eb12-11ee-95d0-87774f69a715 --force 2024-03-26T02:15:24.156 INFO:tasks.cephadm:Removing cephadm ... 2024-03-26T02:15:24.157 DEBUG:teuthology.orchestra.run.smithi019:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T02:15:24.172 DEBUG:teuthology.orchestra.run.smithi069:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T02:15:24.187 INFO:tasks.cephadm:Teardown complete 2024-03-26T02:15:24.188 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-26T02:15:24.199 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-26T02:15:24.200 DEBUG:teuthology.orchestra.run.smithi019:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T02:15:24.215 DEBUG:teuthology.orchestra.run.smithi069:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T02:15:24.226 INFO:teuthology.orchestra.run.smithi019.stderr:bash: line 1: ntpq: command not found 2024-03-26T02:15:24.231 INFO:teuthology.orchestra.run.smithi019.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T02:15:24.231 INFO:teuthology.orchestra.run.smithi019.stdout:=============================================================================== 2024-03-26T02:15:24.231 INFO:teuthology.orchestra.run.smithi019.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 25 -93us[ -93us] +/- 79ms 2024-03-26T02:15:24.231 INFO:teuthology.orchestra.run.smithi019.stdout:^* hv02.front.sepia.ceph.com 3 6 377 25 -398us[ -379us] +/- 60ms 2024-03-26T02:15:24.231 INFO:teuthology.orchestra.run.smithi019.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 25 +631us[ +651us] +/- 79ms 2024-03-26T02:15:24.231 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:15:24.241 INFO:teuthology.orchestra.run.smithi069.stderr:bash: line 1: ntpq: command not found 2024-03-26T02:15:24.246 INFO:teuthology.orchestra.run.smithi069.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T02:15:24.246 INFO:teuthology.orchestra.run.smithi069.stdout:=============================================================================== 2024-03-26T02:15:24.246 INFO:teuthology.orchestra.run.smithi069.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 26 -56us[ -56us] +/- 79ms 2024-03-26T02:15:24.246 INFO:teuthology.orchestra.run.smithi069.stdout:^* hv02.front.sepia.ceph.com 3 6 377 30 -380us[ -368us] +/- 60ms 2024-03-26T02:15:24.246 INFO:teuthology.orchestra.run.smithi069.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 +651us[ +651us] +/- 79ms 2024-03-26T02:15:24.246 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:15:24.246 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-26T02:15:24.258 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-26T02:15:24.259 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-26T02:15:24.281 DEBUG:teuthology.orchestra.run.smithi019:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T02:15:24.365 DEBUG:teuthology.orchestra.run.smithi069:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T02:15:24.443 DEBUG:teuthology.orchestra.run.smithi019:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-26T02:15:24.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:15:24.473 DEBUG:teuthology.orchestra.run.smithi069:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-26T02:15:24.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:15:24.503 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-26T02:15:24.514 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-26T02:15:24.546 INFO:teuthology.task.internal:Duration was 2116.490650 seconds 2024-03-26T02:15:24.547 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-26T02:15:24.576 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-26T02:15:24.576 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T02:15:24.579 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T02:15:24.619 INFO:teuthology.orchestra.run.smithi019.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T02:15:24.621 INFO:teuthology.orchestra.run.smithi069.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T02:15:25.059 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-26T02:15:25.059 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi019.front.sepia.ceph.com 2024-03-26T02:15:25.059 DEBUG:teuthology.orchestra.run.smithi019:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T02:15:25.117 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi069.front.sepia.ceph.com 2024-03-26T02:15:25.118 DEBUG:teuthology.orchestra.run.smithi069:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T02:15:25.145 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-26T02:15:25.145 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T02:15:25.159 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T02:15:25.214 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-26T02:15:25.214 DEBUG:teuthology.orchestra.run.smithi019:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T02:15:25.228 DEBUG:teuthology.orchestra.run.smithi069:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T02:15:39.226 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-26T02:15:39.238 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-26T02:15:39.238 DEBUG:teuthology.orchestra.run.smithi019:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T02:15:39.265 DEBUG:teuthology.orchestra.run.smithi069:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T02:15:39.297 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-26T02:15:39.351 DEBUG:teuthology.orchestra.run.smithi019:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T02:15:39.353 DEBUG:teuthology.orchestra.run.smithi069:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T02:15:39.376 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern = core 2024-03-26T02:15:39.382 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = core 2024-03-26T02:15:39.412 DEBUG:teuthology.orchestra.run.smithi019:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T02:15:39.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:15:39.466 DEBUG:teuthology.orchestra.run.smithi069:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T02:15:39.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:15:39.482 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-26T02:15:39.494 INFO:teuthology.task.internal:Transferring archived files... 2024-03-26T02:15:39.495 DEBUG:teuthology.misc:Transferring archived files from smithi019:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621920/remote/smithi019 2024-03-26T02:15:39.495 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T02:15:39.570 DEBUG:teuthology.misc:Transferring archived files from smithi069:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621920/remote/smithi069 2024-03-26T02:15:39.570 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T02:15:39.614 INFO:teuthology.task.internal:Removing archive directory... 2024-03-26T02:15:39.614 DEBUG:teuthology.orchestra.run.smithi019:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T02:15:39.617 DEBUG:teuthology.orchestra.run.smithi069:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T02:15:39.664 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-26T02:15:39.677 INFO:teuthology.task.internal:Not uploading archives. 2024-03-26T02:15:39.677 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-26T02:15:39.717 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-26T02:15:39.718 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T02:15:39.723 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T02:15:39.735 INFO:teuthology.orchestra.run.smithi019.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 02:15 /home/ubuntu/cephtest 2024-03-26T02:15:39.739 INFO:teuthology.orchestra.run.smithi069.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 02:15 /home/ubuntu/cephtest 2024-03-26T02:15:39.740 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-26T02:15:39.783 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-26T02:15:39.857 INFO:teuthology.nuke:Checking targets against current locks 2024-03-26T02:15:39.887 DEBUG:teuthology.nuke:shortname: smithi019 2024-03-26T02:15:39.888 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T02:15:39.911 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi019.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621920', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 01:31:52.202525', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNARUXwtGAtTikQf6mQLs0++GTP3mUSGmJkGxI2L3YK/8DemFBOgWGq4CI1bAzv3ro4KJ6LTu9PWtnI8awUBNCg='} 2024-03-26T02:15:39.918 DEBUG:teuthology.nuke:shortname: smithi069 2024-03-26T02:15:39.918 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T02:15:39.939 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621920', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 01:31:52.204192', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN/SSNXMqbRRu6yK4qoW+u8KiihIo/Fuyw+tGvc1HH69D5i259F5aRCmQNaJakPSwhwY2Q5EeEJNe6sNQ9clQXI='} 2024-03-26T02:15:39.974 INFO:teuthology.orchestra.console.smithi019:Power off 2024-03-26T02:15:39.974 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T02:15:39.990 INFO:teuthology.orchestra.console.smithi069:Power off 2024-03-26T02:15:39.990 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T02:15:40.002 DEBUG:teuthology.orchestra.console.smithi019:power off output: Chassis Power Control: Down/Off 2024-03-26T02:15:40.003 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:15:40.017 DEBUG:teuthology.orchestra.console.smithi069:power off output: Chassis Power Control: Down/Off 2024-03-26T02:15:40.018 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:15:40.030 DEBUG:teuthology.orchestra.console.smithi019:check power output: Chassis Power is on 2024-03-26T02:15:40.042 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-03-26T02:15:44.031 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:15:44.042 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:15:44.141 DEBUG:teuthology.orchestra.console.smithi019:check power output: Chassis Power is on 2024-03-26T02:15:44.152 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-03-26T02:15:48.146 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:15:48.155 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:15:48.257 DEBUG:teuthology.orchestra.console.smithi019:check power output: Chassis Power is on 2024-03-26T02:15:48.266 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-03-26T02:15:52.262 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:15:52.271 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:15:52.374 DEBUG:teuthology.orchestra.console.smithi019:check power output: Chassis Power is off 2024-03-26T02:15:52.382 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is off 2024-03-26T02:15:52.475 INFO:teuthology.orchestra.console.smithi019:Power off completed 2024-03-26T02:15:52.483 INFO:teuthology.orchestra.console.smithi069:Power off completed 2024-03-26T02:15:52.587 INFO:teuthology.run:Summary data: description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} duration: 2116.4906504154205 failure_reason: '"2024-03-26T01:50:42.933185+0000 mon.smithi019 (mon.0) 504 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-03-26T02:15:52.587 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T02:15:52.676 INFO:teuthology.run:FAIL